How is the effect of one dollar spent on a community development project measured, be it an affordable home, subsidized small business loan, financial coaching, or homeownership counseling? Community development struggles with outcome measurement, as do many socially driven fields. Proper measurement takes a specialized expertise. There’s also a lot of room for disagreement on how and what to measure. And measuring is expensive.

 

The cost and time involved in collecting effective success measures are significant, but the price of these efforts cannot be seen merely as a cost – it is an essential investment. Every piece of data that is tracked should be viewed as a down payment on future funding and better outcomes. These data will demonstrate in concrete terms how effective a program is and show stakeholders exactly what a program is accomplishing. But the inherently local nature of service provision provides a validity challenge. Problems are locally unique, as are the solutions. The data measurement naturally follows suit and is framed in a local context. This severely decreases the external validity of any conclusions; making comparisons across localities, at best, invalid and, at worst, irrelevant.

 

Into this environment NACEDA, the National Alliance of Community Development Associations, with its 42 state and regional association members in 28 states stepped in to assess its members’ work in financial empowerment. By their very nature, these associations are aggregators and resource providers who enable their own local organizations to meet the needs of the population.

 

NACEDA wanted to know how its members are measuring financial empowerment programs at the modular level (i.e., are they measuring outputs or outcomes) and also at the metric level (i.e., are they using homeownership rates, credit scores, or other accomplishment-oriented metrics). At the modular level, members were using a little bit of both. Some were simply measuring output metrics, such as the number of visits to a financial coach. Others measured accomplishment oriented outcomes, such as the improvement of a credit score or decreased reliance on alternative financial services.

 

The NACEDA assessment sheds light on these critical enablers in the financial empowerment value chain and tried to answer a broader measurement question, “how does the field of financial empowerment balance the legitimate desire for standardized data with the need for local solutions and implementation?” This essay proposes three key principles based on information collected from the NACEDA assessment and informed by emerging thought on program evaluation and community development.

 

The Field Needs a Common Language

Before any substantive discussion about financial empowerment can begin the disparity in terminology in the field must be addressed. Without a common understanding of one another’s work, organizations will never reach the point of being able to discuss issues and solutions across organizationally boundaries. NACEDA’s assessment found a wide variance in the terms used.

 

Leading organizations in financial empowerment must advance the use of standard terminology. This will afford organizations participating in financial empowerment the means to effectively communicate about their work. Consensus and understanding can then be built by recognizing issues thereby establishing a common idiom for discussion and fostering a culture around financial empowerment.

 

NACEDA found the need for common language during its 2014 assessment. Survey participants were asked to provide the term they though best described this statement:

 

“Strategies to help families with low and moderate incomes stabilize their finances, prepare for their financial future, budget soundly, improve creditworthiness, build assets, and reduce debt.”

 

Many respondents agreed that financial empowerment and asset building are terms that could be used to describe the above mentioned strategies, though they were far from the only terms. Some state and regional associations labeled this definition “economic opportunity,” others used “assets and opportunities” or “credit building.”

 

Standardizing terminology is foundational to the advancement of financial empowerment.

 

Measurement Should Take Place at the State and Regional Level

Broad execution of financial empowerment programs requires tailoring of national solutions to distinct local needs. A local organization’s ability to tailor solutions to their own communities’ needs is the essence of local organizations’ work. Sometimes this tailoring of metrics creates difficulty (i.e. when metrics aren’t relevant outside a given locality; external validity), but overall tailoring makes programs relevant.

State or regionally-based measurement and data collection provides tailored solutions to local needs while bridging the gap to national programs. The NACEDA assessment found that regional or state based associations are balancing the need for standardized data with the need for local solutions and implementation. The detail and sophistication of this measurement varied, but the fact that these organizations recognized the importance of measurement - and are acting on it - is vital. Building on this existing work in a substantive way will advance outcome measurement that is both externally and internally valid.

 

Entities Performing the Measurement Need Access to Technical Expertise on Outcome Measurement

Organizations must prioritize measuring outcomes over outputs and recognize the long-term value in tracking these metrics. But often the specific expertise needed to design effective measurement programs is not available at the local, or even regional level. Skilled individuals that know the difference between outcomes and outputs are an essential resource for these organizations.

 

While, for the most part, any measurement is better than no measurement, organizations should strive to measure outcomes not outputs. The hours an individual spends in financial empowerment training demonstrates an expenditure in time by both the individual and the organization (output), but the increase in that individual’s savings rate indicates a change in behavior (outcome). The number of potential home buyers attending a first-time home buyer workshop (output) is easy for the organization to track, but an increase in attendees’ credit scores (outcome) shows that the workshop had the intended effect. Changing behavior is our goal, not simply training. As such, outcomes should be the primary measure.

 

A common language, effective measurement at the regional level, and access to technical expertise will continue to advance financial empowerment and meet the needs of our communities.