Evaluation of the Council of Canadian Academies

From: Innovation, Science and Economic Development Canada

March 16, 2018

Table of contents

• Executive summary
• Background
• Methodology
• Findings: Relevance and Continued Need
• Findings: Achievement of Expected Outcomes
• Findings: Efficiency and Effectiveness
• Conclusions
• Recommendations
• Appendices
• Endnotes
• Management Response and Action Plan
• Summary of the Evaluation of the Council of Canadian Academies (CCA)

---

Executive summary

The evaluation assessed the relevance and performance of the CCA from 2005-06 to 2016-17 based on qualitative and quantitative research.

The evaluation found that the CCA responds to a need for independent, objective, and transparent scientific knowledge to support evidence-based decision-making. Requests for CCA assessments have been steady since its inception.

In terms of performance, evidence suggests that the CCA has achieved its expected results. The CCA has produced credible, independent and evidence-based assessments for the federal government. Although the CCA conducts some awareness activities to highlight its assessment findings, more can be done in terms of dissemination of its assessments. There is evidence that the CCA assessments funded by ISED have supported decision-making. However, the results and impacts of assessments are difficult to measure and are not tracked by ISED.

ISED's proposal submission and approval process is effective at supporting the steady conduct of CCA assessments on behalf of the federal government.
However, it is both lengthy and unpredictable and federal departments/agencies have recently been submitting more proposals directly to the CCA for urgent assessments using their own funding.

Overall, the evaluation found that CCA program delivery is efficient and effective.
The CCA's use of volunteer experts helps drive down the cost of its assessments.  Assessment and administration costs are in line with expectations, and the organization has improved the leveraging of funding from other sources in recent years. Timelines to complete an assessment are largely consistent with targets.

In terms of the CCA's governance and operations, it has made improvements in recent years, which have contributed to efficiency and effectiveness gains in the production of assessments.

Background

• Program Description and Objectives
• CCA Organizational Structure
• Requests and Funding for CCA Assessments

Program Description and Objectives

Convening experts. Assessing evidence. Informing decisions.

Objective of the CCA

The Council of Canadian Academies (CCA) is a not-for-profit organization that was created in 2005 to produce assessment reports that cover a broad range of science, incorporating the natural, social, and health sciences, as well as engineering and the humanities.

The objective of the CCA is to enhance Canadians' access to the best available scientific knowledge on pressing issues by providing credible, independent assessments that can inform debate, discussion, and decision-making.^{Footnote 1} The assessments are conducted by multidisciplinary panels of volunteer experts from across Canada and abroad.

About the CCA Assessments

The overarching goal of the CCA assessments is to evaluate and present the best available evidence on complex issues where the science may be challenging to understand, conflicting, or difficult to gather. The assessments strive to identify emerging issues, gaps in knowledge, Canadian strengths, and international trends and practices. The assessments do not provide recommendations and instead, formulate conclusions based on existing scientific knowledge.

Stakeholders who request assessments do not have a role or influence in the conduct of these assessments, and do not have access to them until they are completed and made publicly available.

Previously, the CCA offered three types of assessments: standard, streamlined and workshops. However, in 2016-17, the streamlined and standard assessments were collapsed into one overarching category, as it was found that streamlined assessments require similar resources in terms of volunteer experts, meetings, and time to gather the information and assemble the final product.

CCA Organizational Structure

The CCA is led by a 12-member Board of Directors, which is responsible for setting the strategic direction of the CCA. It is supported in its governance by its three founding Academies. Each Academy appoints two members to the Board; these appointments can be Fellows or senior decision-makers of the Academies. These six members then together appoint two Directors from the general public. The remaining four Directors are proposed by the Minister of Innovation, Science and Economic Development and are formally appointed by the Board. A representative from ISED participates on the Board as an observer.^{Footnote 3}

The Board also appoints a Scientific Advisory Committee (SAC) that is composed of experts who have earned credibility among their peers and who have developed a national or international reputation in their field of research. The SAC provides expert advice on the suitability of potential subjects for assessments and input on the selection of expert panelists and reviewers.

Requests and Funding for CCA Assessments

To date, ISED has launched a total of 14 calls for proposals.

Requests for CCA Assessments

The CCA receives requests for assessments from federal departments and agencies either through ISED's call for proposals or through requests made directly to the CCA. It is also encouraged to seek assessment requests from other levels of government, non-governmental organizations, and the private sector.

ISED's Submission and Approval Process

ISED funds CCA assessments on behalf of the entire federal government. Assessments are selected through a process that begins with a call for proposals. To date, 40% of the completed assessments funded through ISED have been for departments and agencies other than ISED.

ISED's Science Programs and Partnerships Branch is responsible for the implementation of the contribution to the CCA, the ongoing management and oversight of the funding for assessments, the launch of the call for proposals, and the proposal submission and approval process.

Proposals submitted to ISED by federal departments and agencies are first reviewed and recommended for approval by ISED based on the following criteria before referral to the CCA:

• the proposal is relevant to the policy agenda of the department/agency and Canada;
• the assessment topic is timely and the timeframe for the assessment is consistent with the needs of the department/agency of Canada;
• the assessment topic has been coordinated among relevant departments/agencies and external stakeholders; and
• the value provided by the CCA is unique.^{Footnote 4}

At least 3 assessment requests were submitted to ISED by departments and agencies during each call for proposal.

Methodology

• Evaluation Objectives, Scope and Approach
• Data Collection Methods

Evaluation Objectives, Scope and Approach

The evaluation covered the period from 2005-06 to 2016-17.

Evaluation Objectives and Scope

The objectives of the evaluation were to examine the relevance, performance, efficiency and effectiveness of the CCA over time.

ISED's Audit and Evaluation Branch (AEB) undertook an in-house evaluation of the CCA. The evaluation covered key evaluation issues of relevance and performance.^{Footnote 9} The evaluation focused on:

• the relevance of the CCA;
• the activities and performance of the CCA between 2005-06 and 2016-17;
• the efficiency and effectiveness of ISED's program delivery model and oversight of the CCA; and
• the efficiency and effectiveness of the CCA's operations and governance.

Evaluation Approach

The relevance of the CCA was assessed based on the extent to which there is a continued need for the CCA and its assessment work. The evaluation assessed the performance of the CCA against its logic model (see Appendix B). In assessing the efficiency and effectiveness of ISED's funding, the evaluation focused on program delivery and ISED's oversight. It also assessed the efficiency of the CCA's operations in terms of timeliness and the cost of assessments, as well as recent improvements to the CCA's governance structure.

Data Collection Methods

The evaluation was based on qualitative and quantitative research methods. Data limitations are described in Appendix C.

Data Analysis

An analysis of the CCA's performance, administrative and financial data was performed. ISED's program data was also included in the analysis.

Bibliometric Analysis

Bibliometric analysis included an analysis of citations (e.g., direct and indirect citation counts based on Scopus and Google Scholar databases). It also included an analysis of authors (e.g., geography and affiliations) and publication sources (e.g., journals, books, patents, conferences).

Literature and Document Review

A review of academic literature related to the mechanisms and approaches used to deliver evidence-based scientific knowledge to support government decision-making was performed. In addition, federal government priority setting documents, ISED program and foundational documents, and CCA corporate and governance documents were also reviewed.

Interviews

A total of fourteen group and individual interviews were conducted with ISED program staff, federal departments and agencies that have requested assessments directly from the CCA, CCA staff , the Academies, members of the Board of Governors, and the Science Advisory Committee.

Case Studies

Seven CCA assessments were selected for the case studies (State of S&T 2006, State of S&T 2012, Conducted Energy Weapons 2013, Shale Gas 2013, Science Culture 2014, STEM Skills 2015, and Regenerative Medicine 2017). Goss Gilroy Inc. was contracted to conduct these case studies, which included a document review (e.g., sponsor surveys, lessons learned documents), an analysis of CCA performance data (e.g., citations in federal documents, media and academia), and twenty-three interviews.

International Comparative Analysis

Four international organizations were selected for the comparative analysis (US National Academy of Sciences, UK Royal Society, Australian Council of Learned Academies, and the German National Academy of Sciences Leopoldina). Goss Gilroy Inc. was contracted to perform a comparison of the CCA's operations and governance to international organizations with similar mandates (i.e., as independent sources of expert scientific assessments). As part of this analysis, three interviews were conducted with international organizations.

Findings: Relevance

Continued need for scientific knowledge and CCA assessments

Finding: The CCA addresses a need for independent, objective, and transparent scientific knowledge to support evidence- based decision-making. Requests for CCA assessments have been steady since its inception.

Need for independent, objective, and transparent scientific knowledge

A large body of literature has identified a global need for independent, objective, and transparent mechanisms to deliver scientific knowledge that can be used to support governments in evidence-based decision-making.^{Footnote 12Footnote 13} The body of literature also emphasized the roles and responsibilities of expert bodies, such as national academies, which is the model used by the CCA, in evidence-based decision- making. A scan of the CCA assessments, coupled with interviews, found that the CCA provides publicly available, independent, objective assessments of the current state of scientific knowledge on a wide range of topics to support evidence-based decision-making and educate Canadians.

Need for scientific knowledge in federal decision-making

The mandate letter to ISED's Minister of Science emphasizes that the federal government believes that good scientific knowledge should inform decision-making. It also sets out an overarching goal for the Minister of Science to support scientific research and the integration of scientific considerations in the federal government's investment and policy choices.^{Footnote 14} Budget 2017 highlighted the Government of Canada's commitment to an evidence-based approach to decision-making. It also cited the CCA, further substantiating the fact that its assessments are supporting federal government decision-making.^{Footnote 15}

Need for CCA assessments by federal departments and agencies

A review of the total number of proposals for CCA assessments submitted by federal departments and agencies since 2005-06 revealed that there has been steady demand across the federal government. Specifically, 32 of the 36 completed CCA assessments, or 89%, were funded by the Government of Canada. Of this total, 30 assessments were funded through ISED's submission and approval process and two were funded by departments/agencies through a direct request to the CCA. Most interviewees suggested that the demand for CCA assessments will continue to grow given the federal government's priority for credible scientific knowledge to support evidence-based decision-making.

Findings: Achievement of Expected Outcomes

• Producing credible, independent, evidence-based assessments
• Increasing visibility and awareness
• Supporting discussion, debate, public policy development and evidence-based decision-making

Producing credible, independent, and evidence-based assessments

Finding: The CCA uses a comprehensive assessment methodology to ensure that its assessments are credible, independent, and evidence-based. The credibility of the CCA is also demonstrated by the volume of citations of its assessments across the world.

The Council's Assessment Lifecycle Methodology

The CCA uses a rigorous assessment methodology to ensure credibility. This process is referred to as the Council's Assessment Lifecycle Methodology (CALM) and it includes requirements and guidelines for all phases of an assessment:

• selection and review of assessment questions, independent panelists and reviewers;
• conduct of an assessment;
• peer review and anti-plagiarism verification; and
• report release and dissemination.^{Footnote 17}

It was stated in several interviews that the CALM is reviewed and revised regularly to ensure that the assessment process remains rigorous and credible.

Volume of direct and indirect citations of CCA assessments

The credibility of the CCA and its assessments is also demonstrated by the number of citations of its assessments. Bibliometric analysis^{Footnote 18} found that 31 of 36, or 86%, of all CCA assessments completed as of 2016-17 were cited by authors in their respective scientific fields using at least one publishing avenue (e.g., journal, book, patent, or conference). This translates to over 500 total direct citations of CCA assessments (i.e., citation counts of publications that had referenced a CCA assessment) and over 4000 indirect citations (i.e., citation counts of publications that had referenced a publication that cited a CCA assessment).

Citations across the world

All 26 of the ISED-funded standard assessments between 2005-06 and 2016-17 have been cited by authors in their respective scientific fields across 43 countries using at least one publishing avenue.

Increasing visibility and awareness

Finding: While the CCA conducts some awareness activities to highlight its assessment findings, the dissemination of its assessments is limited.

Visibility and awareness of the CCA across the federal government

An analysis of program data found that at least 25 federal departments and agencies have submitted at least one request for a CCA assessment through ISED's call for proposals since its inception, suggesting awareness of the CCA. Given that the level of awareness was measured by the total number of proposals submitted by individual federal departments and agencies through ISED's call for proposals, the actual number of departments and agencies that are aware of the CCA may be higher. Interviews also revealed that, within the federal government, the CCA is largely known through word-of-mouth of scientific staff.

Visibility, awareness and dissemination activities of the CCA

Interviews, along with a review of CCA documents, revealed that the CCA's visibility and awareness activities include planning a release for each completed assessment, identifying stakeholders that would be interested in the assessment and disseminating it to a targeted distribution list.^{Footnote 20} The CCA produces videos, infographics, and snapshot summaries to communicate the findings in its assessments. It also does some media and social media outreach, but the extent to which this is done is inconsistent between assessments as interviews indicated that these activities are dependent on resource availability. It was also found from both the review of documents and interviews, that there were instances when the CCA was approached to disseminate its findings and engage in discussions with federal government officials, but this has been inconsistent between assessments. It should be noted that deliverables related to knowledge dissemination are not a requirement in the 2015 funding agreement for CCA assessments.

Recommendation: ISED, in consultation with the CCA, should develop a coordinated approach to improve the dissemination of CCA assessments.

Supporting discussion, debate, public policy development and evidence-based decision-making (1 of 2)

Finding: There is evidence that the CCA assessments have provided decision-makers with independent scientific information. However, the direct results and impacts of CCA assessments are difficult to measure and are not tracked by ISED.

The case studies found that assessments have supported discussion, debate, public policy development and evidence-based decision-making, as shown in the illustrative examples below.^{Footnote 21}

Supporting discussion, debate, public policy development and evidence-based decision-making (2 of 2)

Documenting the results and impacts of CCA assessments

ISED improved its proposal submission template in 2015-16 to include a description of the relevance and expected outcomes/impacts of the assessment. Specifically, federal departments and agencies are required to explain how the proposal is relevant to the federal government's and sponsoring department/agency's policy agenda; the expected socio- economic benefits; and how the assessment could inform future priorities and activities.

However, ISED does not require federal departments/agencies to report back on the results and impacts of CCA assessments, nor does it track the results and impacts of completed assessments that it funds through its call for proposals (though it is not required under the 2015 funding agreement). The results and impacts of assessments are reported only through an optional Sponsor Feedback Survey that is conducted by the CCA one year after an assessment has been completed.

Recommendation: ISED should, in collaboration with the sponsoring federal department/agency, strengthen its tracking of the results and impacts of ISED-funded CCA assessments.

Findings: Efficiency and Effectiveness

• ISED Program Oversight
• CCA Program Delivery
• CCA Governance and Operations

Efficiency and Effectiveness: ISED Program Oversight (1 of 2)

Finding: ISED has established a proposal submission and approval process. However, the process is lengthy and unpredictable. Further, federal departments/agencies are submitting more proposals directly to the CCA for urgent and time-sensitive assessments using their own funding.

ISED's proposal submission and approval process

ISED has a 12-step proposal submission and approval process in place to fund CCA assessments on behalf of the federal government (see Appendix D). It has developed a guidance document^{Footnote 23} for federal departments and agencies on the proposal submission and approval process through which ISED refers assessment topics to the CCA. This guidance document describes the:

• criteria to be applied in determining whether a topic is suitable for an assessment by the CCA;
• types of assessment products that the CCA can produce;
• expected timeframes for completion; and
• steps and key dates of the process through which proposals are developed, refined and referred to the CCA within the ISED call for proposals.

The primary advantage of ISED's approach is that it allows for consistent and centralized oversight of the funding program and the proposal submission and approval process.

Timelines and length of the proposal submission and approval process

Interviews found that ISED's proposal submission and approval process for CCA assessment requests is lengthy, lasting approximately 12 months.

Further, the timelines associated with individual proposals at each step of the submission and approval process are not tracked by ISED. This makes it difficult to identify bottlenecks and improve the efficiency of the proposal submission and approval process.

Efficiency and Effectiveness: ISED Program Oversight (2 of 2)

Predictability of the calls for proposals and the urgency of assessments

 A review of CCA documents^{Footnote 24} and interviews confirmed that since 2016- 17 at least eight federal departments and agencies have approached the CCA expressing interest in submitting an assessment request directly to them, up significantly from the previous ten-year period (2005-06 to 2015-16) where only two federal departments had funded their own assessments. Further, ISED has not launched a call for proposals since 2015-16 given that the CCA has already reached its target of 18-20 assessments funded by ISED under the 2015 funding agreement.^{Footnote 25}

Other than the lack of a call for proposals, interviews confirmed that reasons for submitting requests directly to the CCA include the urgency of an assessment, and the length and certainty of the approval and planning process. Where there is an immediate need for CCA assessments, federal departments require certainty that their topic will be approved. In these cases, they interact with the CCA directly to refine the question and scope, and plan the assessment. It was also found from interviews that the sign-off process for assessment requests that were submitted directly to the CCA were less onerous, with the planning and approval process being as short as a few months. Interviews also noted that certainty on the topic enables the CCA to take immediate action on resourcing needs and operational planning. These interviews emphasized the need for flexibility in the process for time-sensitive assessment requests.

Recommendations:

ISED should review its submission and approval process to simplify and better respond to the timelines and needs for scientific knowledge by the federal government. Going forward, consideration should be given to a process which would increase predictability for both the CCA and the federal government.

ISED should develop service standards for key steps in the proposal submission and approval process.

Efficiency and Effectiveness: CCA Program Delivery (1 of 2)

Finding: The CCA's use of volunteer experts helps drive down the cost of its assessments. Assessment and administration costs are in line with expectations, and the organization has improved the leveraging of funding from other sources in recent years. Timelines to complete an assessment are largely consistent with targets.

Allocation of expenditures

An analysis of CCA financial data found that total CCA expenditures from 2005-06 to 2016-17 was $46.5M, with an annual average of 3.9M.^{Footnote 26} Roughly 66% ($30.6M) of total spending was spent on the conduct, production and dissemination of assessments. This is in line with the funding agreement which specifies that at least 64% of ISED's funding should be spent directly on assessment-related activities.^{Footnote 27} Indirect expenditures, such as overhead and administration, took up the remaining 34% ($16.0M), consistent with the funding agreement.^{Footnote 28}

Use of volunteer experts in the conduct of assessments

The CCA relies on volunteer experts for its assessment work, which is consistent with similar international organizations. An analysis of the CCA's volunteer data found that there are, on average, 13 panelists and 11 reviewers per assessment. The CCA has estimated that the average time that each panelist spends on assessment work is 24 days, while each reviewer spends 2.5 days. The CCA also estimated that the compensation for an expert volunteer would be $1200/day. With these estimates, the average contribution of volunteers is equivalent to $407,400 per assessment.^{Footnote 29}

As such, the use of volunteer experts help drive down the cost of CCA assessments. With the average cost of an ISED-funded standard assessment calculated at $1.43M, the cost would be about $1.8M in the absence of volunteer experts.^{Footnote 30}

Efficiency and Effectiveness: CCA Program Delivery (2 of 2)

Completion times for CCA assessments

Overall, completion times for ISED-funded CCA assessments are largely in line with targets. For standard assessments, the CCA took an average of 27 months to complete, slightly more than the completion target of 24 months. For streamlined assessments and workshops, the average completion time fell within the expected timeframe of 12-18 months and 6 months, respectively.

As Figure 1 shows, the average time needed to complete an ISED-funded standard assessment varies, from a minimum of 19 months to a maximum of 34 months.^{Footnote 32} These factors are impacted by the complexity and scope of an assessment, which can further increase the completion time. A review of assessments, along with interviews, found that the completion times vary based on complexity of topic and scope. Factors that contribute to the length of time to complete an assessment are consensus building among multidisciplinary panelists, and the review and consideration of all the comments provided to the panel from the reviewers. As the CCA improved the rigour of its assessment methodology over the years, it took longer, on average, to complete ISED-funded standard assessments with complex topics and scopes. However, some factors that impact the completion time are outside of the CCA's control (i.e., delays in finalizing an assessment request).

Efficiency and Effectiveness: CCA Governance and Operations

Finding: The CCA has made improvements to its governance and operations in recent years, which have contributed to efficiency and effectiveness gains in the production of assessments.

The role of the Academies in CCA governance

Effective and efficient working relationships between the CCA and the Academies are critical to the production of assessments. In recent years, the CCA has improved its governance by ensuring that the President of each Academy is one of the two Directors appointed. Interviews confirmed that this has improved dialogue between the Directors and simplified internal communication when reporting back to the individual Academies.

In 2016-17, a Collaboration Agreement was established between the CCA and the Academies to further improve the efficiency and effectiveness of the CCA governance.^{Footnote 33} The Agreement serves to clarify and formalize the roles and responsibilities of the Academies and lay out the terms and conditions for collaboration. It also serves to act as a mechanism to recognize the services of the Academies and remunerate them for documented expenses related to eligible activities under the 2015 funding agreement.^{Footnote 34} Eligible activities are activities directly related to supporting, conducting, and producing assessments on topics which are referred by the Minister of Science. The comparative analysis found that national academies within other jurisdictions also play similar roles in the governance of their respective CCA-like organization.

Horizontal oversight of CCA assessments and volunteer expert panels

The CCA has made recent changes to its organizational structure to improve the efficiency and effectiveness of its operations. A Director of Assessments position was created in 2016-17 to allow for consistent and horizontal oversight of all active assessments. This position is responsible for resource allocation and coordination across assessments, overseeing quality standards and report writing. Interviews confirmed that the staffing of this position hascontributed to better resource management and has improved the quality of assessments.

It was also found from both the document review^{Footnote 35} and interviews that the CCA has recently improved the selection process for its volunteer expert panels to ensure that it continues to produce high quality assessments. The vetting process for a Panel Chair was revised to include a review of research and social media profiles to ensure that Chairs not only have the expertise and proven chairing abilities, but also the ability to prepare an objective report. Co-chairs have also been recently introduced which has proven to be an effective and efficient approach when a subject has two well established positions or when a Chair has limited availability.

Improved relationship with the three founding Academies

It was noted from interviews that the relationship with the Academies has improved in the last two years, and has been further strengthened as a result of the signed Collaboration Agreement.

Conclusions

Overall, the CCA continues to be relevant and has achieved its expected outcomes. The program delivery model and CCA governance/operations are effective and efficient. However, there are opportunities to improve the dissemination of CCA assessments, as well as the efficiency and effectiveness of ISED's oversight of the CCA.

Relevance: The CCA addresses a need for independent, objective, and transparent scientific knowledge to support evidence-based decision-making. Requests for CCA assessments have been steady since its inception.

Achievement of Expected Outcomes: The CCA uses a comprehensive assessment methodology to ensure that its assessments are credible, independent, and evidence-based. The credibility of the CCA is also demonstrated by the volume of citations of its assessments across the world. While the CCA conducts some awareness activities to highlight its assessment findings, the dissemination of its assessments is limited. There is evidence that the CCA assessments have provided decision-makers with independent scientific information. However, the results and impacts of CCA assessments are difficult to measure and are not tracked by ISED.

Efficiency and Effectiveness of ISED Program Oversight: ISED has established a proposal submission and approval process. However, the process is lengthy and unpredictable. ISED has established a proposal submission and approval process.
However, the process is lengthy and unpredictable. Further, federal departments/agencies are submitting more proposals directly to the CCA for urgent and time-sensitive assessments using their own funding.

Efficiency and Effectiveness of CCA Program Delivery: The CCA's use of volunteer experts helps drive down the cost of its assessments. Assessment and administration costs are in line with expectations, and the organization has improved the leveraging of funding from other sources in recent years. Timelines to complete an assessment are largely consistent with targets.

Efficiency and Effectiveness of CCA Governance and Operations: The CCA has made improvements to its governance and operations in recent years, which have contributed to efficiency and effectiveness gains in the production of assessments.

Recommendations

The findings of the evaluation led to the following recommendations.

It is recommended that ISED:

1. Develop, in consultation with the CCA, a coordinated approach to improve the dissemination of CCA assessments.
2. In collaboration with the sponsoring federal department/agency, strengthen its tracking of the results and impacts of ISED-funded CCA assessments.
3. ISED should review its submission and approval process to simplify and better respond to the timelines and needs for scientific knowledge by the federal government. Going forward, consideration should be given to a process which would increase predictability for both the CCA and the federal government.
4. Develop service standards for key steps in the proposal submission and approval process.

Appendices

• Appendix A: Assessments from 2005-06 to 2016-17
• Appendix B: Program Logic Model
• Appendix C: Data Limitations
• Appendix D: ISED Proposal Submission and Approval Process

Appendix A: Assessments from 2005-06 to 2016-17 (1 of 2)

Assessments Funded by ISED

Assessments Funded by ISED	Fiscal Year of Completion
1. The State of Science and Technology in Canada, 2006	2006-07
2. Influenza Transmission and Personal Protective Respiratory Equipment	2007-08
3. Energy from Gas Hydrates	2008-09
4. Nanotechnology: Small is Different	2008-09
5. Sustainable Management of Groundwater in Canada	2009-10
6. Better Research for Better Business	2009-10
7. Innovation and Business Strategy: Why Canada Falls Short	2009-10
8. Honesty, Accountability and Trust: Fostering Research Integrity in Canada	2010-11
9. Canadian Taxonomy: Exploring Biodiversity, Creating Opportunity	2010-11
10. Healthy Animals, Healthy Canada	2011-12
11. Integrating Emerging Technologies into Chemical Safety Assessment	2011-12
12. Informing Research Choices: Indicators and Judgment	2012-13
13. The State of Science and Technology in Canada, 2012	2012-13
14. Strengthening Canada's Research Capacity: the Gender Dimension	2012-13
15. Water and Agriculture in Canada: Towards Sustainable Management of Water Resources	2012-13
16. The State of Industrial R&D in Canada	2013-14
17. Aboriginal Food Security in Northern Canada	2013-14
18. Environmental Impacts of Shale Gas Extraction in Canada	2014-15
19. Enabling Sustainability in an Interconnected World	2014-15
20. Improving Medicines for Children in Canada	2014-15
21. Energy Prices and Business Decision Making in Canada: Preparing for the Energy Future	2014-15
22. Leading in the Digital World: Opportunities for Canada's Memory Institutions	2014-15
23. Science Culture: Where Canada Stands	2014-15
24. Policing Canada in the 21st Century: New Policing for New Challenges	2014-15
25. Accessing Health and Health Related Data in Canada	2014-15
26. Some Assembly Required: STEM Skills and Canada's Economic Productivity	2015-16
27. Technological Prospects for Reducing the Environmental Footprint of the Oil Sands	2015-16
28. Health Product Risk Communication: Is the Message Getting Through?	2015-16
29. Understanding the Evidence: Wind Turbine Noise	2015-16
30. Building on Canada's Strengths in Regenerative Medicine	2016-17

Assessments Funded by Other Departments/Agencies

Assessments Funded by Other Departments/Agencies	Fiscal Year of Completion
31. Vision for the Canadian Arctic Research Initiative	2008-09
32. Health Effects of Conducted Energy Weapons	2013-14

Assessments Funded by Other Departments/Agencies

Assessments Funded by Other Departments/Agencies	Fiscal Year of Completion
33. Innovation Impacts: Measurement and Assessment	2013-14
34. Ocean Science in Canada: Meeting the Challenge, Seizing the Opportunity	2013-14
35. Technology and Policy Options for a Low-Emission Energy System in Canada	2015-16
36. Commercial Marine Shipping Accidents: Understanding the Risks in Canada	2016-17

Appendix B: Program Logic Model

Appendix C: Data Limitations

Data Availability: A standard CCA assessment has a target completion timeframe of 24 months and the CCA conducts a voluntary Sponsor Feedback Survey 12 months after the release of an assessment to collect data on assessment impacts. Often, the impacts of an assessment are either difficult to measure or are not observed until several years after its release. For the case studies, between this time period, sponsoring departments and agencies experienced employee turnover and retirements, particularly of those who were initially involved in the proposal submissions for assessments. As a result, the list of potential interview contacts for the case studies was limited.

Potential Bias in Interview Data: Interview participants holding the broadest knowledge of the CCA and its operations are individuals who were active or previous members of the CCA's Board of Governors and/or the CCA's Science Advisory Committee. The opinions and perspectives of these individuals may be slightly biased due to their involvement in the CCA's governance structure, however, this limitation was mitigated through triangulation of findings from other lines of evidence.

 Strength of International Comparative Analysis: Various factors limited the comparative analysis with international organizations. First, there was no organization fully comparable to the CCA in terms of objectives, funding model, operations, governance structure, and size. Second, the data necessary to compare the cost per assessments and overhead costs between the selected international organizations and the CCA was either insufficient or unavailable.

Appendix D: ISED Proposal Submission and Approval Process

Endnotes

Management Response and Action Plan

A - For inclusion in the report

The findings and recommendations of the Council of Canadian Academies Evaluation Report were presented to program management. Management has agreed with the findings included in this report and will take action to address all applicable recommendations by May 31, 2019

B - For follow-up purposes - Detailed actions to address the recommendations in the report