While it is widely recognized that impact measurement is important for investors, the value that impact measurement creates for social enterprises and NGOs is often overlooked. Impact measurement in the social sector currently exists within a largely compliance-based paradigm. Social enterprises and NGOs collect impact data, often at great cost, in order to meet the requirements of their investors, who set those requirements based on the expectations of their limited partners. With pressure to please funders or impress investors, few organizations effectively leverage impact measurement to truly understand how their work impacts those they are trying to serve. Although impact data plays an important role in providing upward accountability and is critical to the impact investing field, it should first and foremost be a tool
for organizations to evaluate and improve their work.

Although a number of organizations have attempted to build impact evaluation metrics and frameworks, there is still no universally accepted standard. This makes it nearly impossible to compare impact across organizations. The lack of universal measurement standards also makes it difficult for NGOs and social enterprises to calculate their impact, as they must often report multiple sets of metrics to please different investors. The impact sector needs a simple but flexible impact measurement framework that provides value for both entrepreneurs and investors.

Over the past semester, we have worked to develop a standardized measurement evaluation template that can be used to collect relevant metrics and quantify the impact on NGOs and social enterprises working to address UN Sustainable Development Goal 7: Affordable and Clean Energy. This template aims to 1) support investors and donors looking to make decisions about how to allocate their resources and 2) help social sector organizations quantify their impact.

We began our research by reading and analyzing existing literature on the energy sector, SDG 7, impact measurement, and impact investing. After developing a deeper understanding of the clean energy landscape, we researched dozens of US-based energy NGOs, tracking the metrics that each organization reported on their websites and in annual reports. Comparing these metrics to each other, and to existing frameworks such as IRIS and GOGLA (The Global Association for the Off-Grid Solar Energy Industry), we identified the most commonly reported energy-related metrics. We also conducted primary research by interviewing experts and professors in the energy and impact investing sectors.

Our research yielded several important insights, which informed our process moving forward. The first is how impact measurement is difficult, and can be a meaningful burden for NGOs and social entrepreneurs, especially those that lack significant resources. As a GIIN report on impact measurement states, “Impact investors must balance the demand for more evidence with the practicality and resources required to actually obtain it.”11 A good impact evaluation template should be as simple as possible for entrepreneurs to complete, while also being detailed enough to provide investors with valuable information.

The second insight is that the NGOs that we researched reported a wide variety of metrics. In order to create a standardized template, we selected only those that we deemed to be the most important in measuring actual impact. This simplification inherently means that some level of detail and nuance will be lost, which is a necessary trade-off in order to allow for cross-sector comparison.

Thirdly, some NGO organizations are not measuring or reporting impact at all. Of those that do, the majority track outputs rather than outcomes, and they often report outputs that are easy to measure (such as social media impressions) while failing to capture important outcomes (such as the results of their lobbying efforts). Similarly, many organizations
have annual reports full of case studies, quotes, and images, but are not reporting aggregated quantitative data. NGOs need to be incentivized to quantify their impact more concretely and concisely. See Appendix I for impact measurement best practices, including tools that organizations can use to quantify the outcomes of policy change.

Another insight is that investors measure what matters to them. A recent GIIN report explains, “Today, impact investors tend to develop individual, idiosyncratic systems and practices that reflect their impact objectives, those of limited partners (LPs) and investees, and the operating environment in which deals take place.” It is perfectly reasonable that investors have different investment priorities and objectives, and that they care about different metrics.

In fact, the Rockefeller Foundation’s Impact Investing Handbook underscores the importance of investors finding their “why” and selecting investing lenses, categories, and measurement techniques that align with their values.5 As we work to build a standardized model, the investor side should allow investors to look at the metrics that matter most to them when making decisions. However, as previously discussed, impact measurement must ultimately balance investors’ needs with a value-driven approach that helps NGOs evaluate and improve their work.

The final insight is that NGOs in the energy sector fall into two broad categories: those focused on direct service (such as distributing solar lanterns) and those focused on policy change (such as those advocating for clean energy tax incentives). Although some organizations engage in both policy change and direct service, these two types of organizations generally have very different activities, metrics, and impacts.

Our impact evaluation framework was inspired by a matrix used by impact investors to evaluate startups. Each of the columns in the matrix corresponds to a specific impact metric, and each organization gets a score based on its impact in that category. For example, an organization that has prevented between 15,000 and 49,999 metric tons of CO2 equivalent in a given year would get a 3 in the greenhouse gas (GHG) emissions prevented column. An organization’s score for each category is multiplied by the associated weight (row 1 of the matrix) and summed into an overall score.

Organizations are also given a score from 1 to 5 based on their total revenue (see Figure 3). To adjust for organizational size, each organization’s overall score is divided by its revenue score to yield a final impact score. We chose to use revenue as a proxy for size because investors and donors are typically interested in how much impact an organization will create with their investment or donation.

One of the largest issues we grappled with throughout our research process was how to compare organizations doing direct service to those working in policy advocacy. After gathering feedback from several impact investors, we concluded that it would not be practical or feasible to compare these vastly different types of work, so we decided to create two separate scorecards: one for organizations primarily focused on direct service, and one for organizations focused on policy advocacy. Organizations that conduct both direct service and policy advocacy work will complete both matrices, and their overall score will be the sum of their scores from both matrices.

The ranges used in the columns of each of the matrices were calculated based on real data collected from 25 NGOs working in the energy sector. (Note that our analysis was limited to organizations legally registered as nonprofits in the United States). To determine the ranges, we calculated percentiles for our impact data in that category: a 1 corresponds to the 20th percentile on a given metric, 2 corresponds to the 40th percentile, and so on. We then rounded and used our judgment when lacking adequate data to create quantifiable levels of impact, from 0-5.

Our comparison-based approach allows investors to benchmark an organization’s performance against others in the field. Moreover, the matrix format lets investors see how an organization is performing on a single metric (so that they can focus on the metrics they care most about) as well as how they are performing overall, relative to their total revenue. These matrices are designed to measure impact on an annual basis; organizations will complete them each year, and investors can compare scores across years to evaluate an organization’s progress over time.

While we began with a long list of metrics, we ultimately included only four metrics in each of our matrices, two “primary metrics” and two “secondary metrics.” If we truly want to measure the impact of SDG 7, we must concentrate on the metrics that actually measure progress towards creating affordable and clean energy. We accomplish this with our primary metrics, GHG emissions prevented and increased access to affordable clean energy. We chose to include other common and meaningful metrics as secondary metrics but to weigh them as half the value of the primary metrics. While these metrics are valuable, they are indirectly related to progress toward SDG 7 and as such carry less weight. Many of the most commonly reported metrics, such as social media impressions, are excluded from our matrix entirely, as we do not believe that they represent actual impact. See the following section for more details about our metrics.

Our scoring works based on the assumption that if an organization cannot quantify its impact, they do not get credit for it. This forces organizations, especially those working in the policy space, to measure and quantify the downstream effects of their work. Direct service and policy organizations are scored on the same primary metrics and the same impact ranges, with the only difference being those policy organizations are measured based on the impact of the policy they help to pass rather than on their direct work.

When evaluating policy-focused organizations, it is important to note that no single organization is ever responsible for the passage of a given policy. However, we believe that collective impact should be incentivized rather than penalized when evaluating impact. Therefore, rather than attempting to measure the relative contribution of a given organization to a policy win (which would be nearly impossible to do accurately), we allow organizations to report the impact of the entire policy.

These matrices are meant to be used in conjunction with qualitative data. As the GIIN notes, “Qualitative data is not viewed as a substitute for hard numbers, but rather an explanation for and complement to the numerical reach or breadth data often provided by investee companies. It not only helps to prove or contextualize these figures but also allows for a more sophisticated understanding of the causal relationship between outputs and outcomes.”11 As we describe in detail below, NGOs will be asked to supply qualitative data along with the quantitative data needed to calculate their impact score. 

Several of the metrics are associated with multiple closely related IRIS metrics, while others have only loose proxies or lack IRIS metrics entirely. To exclude research that does not generate actual impact, we define “major research reports” as those cited by at least one governmental agency, policymaker, or NGO. Rather than reporting GHG emissions prevented, some NGOs report clean energy created (in kWh), as measured by IRIS metrics PI5842 (renewable energy generated for sale), PD1504 (energy capacity of products sold), and PI7623 (energy savings of products sold). In this case, the emissions prevented by the generation of clean energy can be calculated using the U.S. Environmental Protection Agency’s conversion: 7.09 × 10-4 metric tons CO2 prevented/kWh generated.

To make this conversion as easy as possible for NGOs, we link to the EPA’s Greenhouse Gas Equivalencies calculator in our data collection form (which is described in detail below). We chose to include GHG emissions prevented rather than clean energy generated because it can be compared across multiple SDGs, whereas clean energy generated is specific to SDG. While some organizations report both GHG emissions and clean energy created, we chose to measure only one of these metrics to avoid double counting impact.

Additional metrics, including productive hours gained and reduction in deaths and disability-adjusted life-years (DALYs), may be relevant for organizations that serve bottom-of-the-pyramid (Bop) individuals in emerging economies. These metrics were not relevant to the US-based organizations that we researched (for example, nearly everyone in the U.S. already has some form of electricity, so US-based beneficiaries of clean energy NGOs typically do not see a gain in productive hours when they get access to clean energy). Since our research in this project was limited to US-based NGOs, we did not include these metrics in our matrix. However, they may be relevant to ImpactX as it looks to scale globally.

We now look at the case studies and how to apply the framework. From the list of NGOs we compiled during our research phase, we chose five to apply to our template. As our goal is to develop a scalable methodology applicable to a wide range of impact-driven organizations, we intentionally selected a diverse subset of NGOs, with a large variation in size, focus area, geography, and type of impact.

Solar Sister recruits, trains, and supports female entrepreneurs in sub-Saharan Africa to build businesses and bring clean energy to their communities. Rocky Mountain Institute (RMI) works with businesses, policymakers, communities, and other organizations to identify and scale energy system interventions that will cut greenhouse gas emissions and address the climate crisis. NC GreenPower’s mission is to expand public knowledge and acceptance of clean energy technology throughout North Carolina. With a local, community-based approach, they support renewable energy and carbon offset projects and provide grants for solar installations at K-12 schools.

Grid Alternatives builds community-powered solutions to advance economic and environmental justice through renewable energy. Vote Solar advocates for clean energy in legislative and regulatory arenas at the state level, with the goal of building a thriving clean economy with affordable solar energy for all.

Calculations on Vote Solar organization show that in 2020, Vote Solar achieved 20 legislative wins, which led to 60M metric tons of CO2 reductions and increased access to solar energy for over 100M people, giving it a 5 in the first two categories. first two categories. Vote Solar educated 600 people as energy advocates, giving it a 4 in the education category. Vote Solar does not conduct research and does not report any research-related metrics, so it gets a 0 in the research category. Vote Solar’s total revenue in 2020 was $6,973,076, so its revenue score is 4. Unadjusted Score = 5 + 5 + (0.5)(4) + (0.5)(0) = 12. Therefore the final score is 12 / 4 = 3.0.

Vote Solar effectively quantifies the outcomes of its policy advocacy and consequently has a relatively high impact score. Unfortunately, they seem to be the exception rather than the rule. As previously mentioned, most policy-focused organizations are not currently measuring the outcomes of their advocacy efforts, and as such are unable to report the metrics in our matrix. For example, despite having a nearly $44M budget and doing a huge amount of research, policy
advocacy, and sector-building, Rocky Mountain Institute does not quantify the downstream impact of its efforts. As such, they had the lowest score of our selected organizations. We anticipate that this type of missing data will be a significant challenge, especially in the initial years, and we hope that low impact scores will incentivize organizations to quantify their impact more effectively.

Looking at the NGO user experience, as previously discussed, our goal is to make the impact reporting process as simple as possible for NGOs and social enterprises. While our framework and calculation mechanism are structured as a matrix, we recommend a simple questionnaire to collect the data required to complete our matrices. We prototyped this process with a Google form. Responses from the form are pipelined directly into a spreadsheet. In reality, this process could be embedded into ImpactX’s platform: NGOs would complete the form on ImpactX’s website, and their answers could be programmed directly into the relevant impact matrix to calculate an impact score.

To simulate the investor user experience, we created a series of mockups of what the ImpactX platform could look like for investors. Upon accessing the platform, investors can search or filter organizations by SDG, type of impact (direct service vs. policy change), geographic focus, size, and the metrics themselves. Search results are ranked by impact score.

Investors can click on an organization to be taken to their profile page. Each organization’s profile includes basic information such as its name, location, website, and a short description, along with impact scores and revenues for each year of its history, so that investors can see the trajectory of its impact and revenue over time. By clicking on a specific impact score, investors can see the matrix breakdown for that year. Finally, the profile will also include links to the organization’s annual report (if applicable) and financial statements (e.g. 990).

Our framework and template have several important limitations. Our template is meant to be used by investors to compare organizational impact at a high level. It is not a replacement for more in-depth impact reporting, such as an annual report, nor is it a replacement for investor due diligence. Once an investor has narrowed down their organizations of interest, he or she should conduct additional due diligence (including incorporating qualitative data into their decision-making). 

While the impact ranges used in our matrices are based on real data, they need to be validated with larger sample size and one that is representative of the organizations on which they will be used. (For example, if the matrices are intended to be used on NGOs beyond the US or on for-profit social enterprises, data from these types of organizations should be incorporated).

Not everything that matters can be measured, and there is a risk associated with reducing the impact of numbers and people on statistics. As stated previously, the creation of a standardized template necessarily involves simplification, and such simplification can obscure important nuances. Moreover, our ranges are based on quantity rather than quality, and a higher quantity does not always translate into greater impact, especially for our policy-related metrics. For example, one research report that leads to a significant policy change could be more impactful than ten lower-quality reports.

There are mathematical limitations associated with using a discrete 5-point scale. If an organization increases its impact in a certain category (as compared to the previous year) but does not improve enough to move up to the next range (i.e. from a 3 to a 4), its impact score remains the same, and the marginal improvement in impact is not captured. Policy change often takes longer and is harder to quantify than direct service, so there is a risk of it being undervalued. On the other hand, new policies can also create unintended consequences, and these negative externalities are not captured in our impact measurement process. Our framework does not consider the factor of additionality (the impact that would not occur but for the investment), which is an important consideration for some impact investors.

Despite its limitations, our framework provides a simple, focused way to compare the impact among organizations working in the clean energy sector. By condensing metrics into either direct service or policy categories, this framework allows nearly any clean energy NGO to be evaluated. With some refinement and increased data collection, we believe that it can play a key role in influencing ImpactX’s impact measurement strategy moving forward. Additionally, we believe the basic format and premise of our framework could be expanded to evaluate the impact of organizations working towards other Sustainable Development Goals.