Resources

• 1: Introduction
• 2: Foundations
• 2.1: Research Findings: Philanthropic Evaluation Policies
• 3: Government
• 3.1: Guidance
• 3.2: Research Findings: Government Evaluation Policies
• 4: Publications
• 4.1: AJE (2021)
• 4.2: NDE (2018)

1 - Introduction

• A Program Evaluation is the analysis of the effectiveness of a nonprofit effort.
• An Evaluator is a professional who conducts program evaluations. They often have academic and/or professional background in quantitative and/or qualitative social research methods,
• An Evaluation Policy is a document describing how to conduct Program Evaluations. Evaluation policies are often drafted by a government body or philanthropic foundations to standardize the practice of evaluation across a portfolio of nonprofit programs they support.

2 - Foundations

A few foundations have mature, sophisticated evaluation policies.

Many others are much earlier in their journey.

Here we’ll be collecting research on evaluation policies in the foundation sector and provide guidance to help foundation staff develop their own policies.

2.1 - Research Findings: Philanthropic Evaluation Policies

Background

Evaluation policies lay the foundation for virtually every aspect of evaluation practice. Despite their importance to commissioning, planning, implementing, and using evaluations, we know very little about these policies. To advance our understanding of evaluation policy, research-practitioners from the University of California, Los Angeles (UCLA) set out to identify existing evaluation policies in the U.S. philanthropic sector and learn about their content.

What did we study?

To focus this study, the top 50 foundations that invested most heavily in evaluation in 2013 and 2014 were identified using Foundation Center data. All evaluation grants awarded in those 2 years by each foundation were aggregated, resulting in 67 unique foundations. Then, a systematic review of 67 foundation websites was conducted to identify and catalog evaluation policies. Policies were systematically reviewed for insights on what they most, and least, frequently include.

Data set

12 philanthropic evaluation policies were identified.

Findings

Of these evaluation policies, 35% detailed operational topics such as who is responsible for evaluation tasks. 31% of policies were dedicated to technical topics such as prescriptions for evaluation practice, including expectations for study designs and methods. Policies focused less on how to use evaluation findings (12%), goals for evaluation (12%), and capacity for evaluation (3%).

3 - Government

US Federal agencies and programs are facing growing requirements for Evaluation Policies.

Here we’ve collected research and guidance to help evaluation staff develop their own policies.

3.1 - Guidance

As Evaluation Officers and staff craft new or revise existing evaluation policies, here are some questions to consider:

1. Capacity and Resources

• Is there an expressed commitment to building the knowledge and skills needed to commission, conduct, and use evaluations?
• Is there expressed support for providing funding and personnel time for evaluation?

2. Participation

• How are evaluators selected?
• Is there acknowledgement that evaluation is not just the evaluators’ job, but one that extends to others in the organization?
• Who should be involved and in what capacity (ex: planning, implementing, using findings)?

3. Use

• How will findings from evaluations be used to inform actions? Is this clearly stated?
• How will evaluation use be supported?

4. Equity

• Are evaluations designed to surface local context that may impact the generalizability of findings?
• How are evaluations providing value to grantees and the communities they serve?

3.2 - Research Findings: Government Evaluation Policies

Why?

The Foundations for Evidence-Based Policymaking Act of 2018 requires agencies to develop evaluation policies to regulate and guide their practice (H.R. 4174). Evaluation policies lay the foundation for virtually every aspect of evaluation practice. Despite their importance to commissioning, planning, implementing, and using evaluations, we know very little about these policies.

What did we study?

To fill this gap in knowledge, a team of research-practitioners from Claremont Graduate University and the University of California, Los Angeles set out to identify existing evaluation policies in the Federal government and learn about their content. Publicly available evaluation policies established between 2012 and 2019 at the agency or sub-agency level were cataloged. We reviewed their content for insights on what they most, and least, frequently include.

Data set

17 US Federal evaluation policies were examined:

Findings

Of 17 US Federal evaluation policies, 47% detailed procedural topics such as who is responsible for evaluation tasks, and procedures for commissioning and monitoring evaluations.

35% of policies covered technical topics such as principles guiding evaluation practice and prescriptions for evaluation practice, including expectations for study designs and methods.

Fewer policies covered how to use evaluation findings (13%), how to build stakeholder engagement (8%), or create evaluation capacity (5%)

4 - Publications

4.1 - AJE (2021)

Authors: Kinarsky, Alana R. & Christie, Christina A.

Abstract

Since 2007, two taxonomies have been proposed to identify the components of evaluation practice that may be specified in an evaluation policy. Little is known, however, about how these taxonomies align with evaluation policies developed by philanthropic foundations. Through thematic analysis, this paper first compares 12 foundation evaluation policies to the components of the two existing taxonomies. Based on this analysis, an updated taxonomy is proposed, with 11 components to inform evaluation policies in the philanthropic sector and beyond. The 12 foundation policies are then re-examined in relation to the new taxonomy to compare their length, breadth, and depth. The paper concludes with implications for practice and suggestions for future research.

Full article here

Cite as:

Kinarsky, A. R., & Christie, C. A. (2021). Analysis of Evaluation Policies in the Philanthropic Sector. American Journal of Evaluation.

Supplemental data

• Primary sources: evaluation policies from 12 large foundations

4.2 - NDE (2018)

Author: Kinarsky, Alana R.

Abstract:

Foundations annually distribute over $60 billion to nonprofit organizations. In exchange for funding, nonprofits are often required to evaluate their work by collecting and sharing data that demonstrates their progress and impact. Accordingly, foundations and nonprofits represent a meaningful influence on the evaluation market. The present chapter first describes the historical development of foundations, their important role in society, and their use of evaluation. Informed by this historical account; the chapter then provides an overview of foundation spending on evaluation, detailing the number of evaluation‐related grants and funds allocated by year and by select foundations. In conclusion, the chapter considers implications for the profession and practice of evaluation and offers suggestions for future areas of research.

Full article here

Cite as

Kinarsky, A.R. (2018). The evaluation landscape: U.S. foundation spending on evaluation. New Directions for Evaluation, 160, 81–96.