Once an organization has identified the social capital elements in its programs and specified pathways through which its activities contribute to social capital formation, it can formulate plans to evaluate its social capital impact. Somewhat artificially, we break down this process into steps 4 and 5: Step 4 involves the basic design questions, such as the study's timeframe and the choice of study populations; Step 5 concerns the nuts and bolts of fielding a survey-based evaluation. You can also access a list of social capital survey questions intended to help organizations develop their own survey instruments.

STEP 4: Designing the Evaluation

The design of the evaluation hinges on answering a sequence of questions discussed below. The answers to these questions flow directly from the earlier discussion of what the program is trying to do.

WHO IS THE STUDY GROUP? (expand)

HOW CAN WE BEST IMPLEMENT A BASIC "BEFORE AND AFTER" STUDY FRAMEWORK? (expand)

SHOULD WE USE A COMPARISON GROUP? AND, IF SO, WHO? (collapse)

Task 4: Determine what comparison group, if any, will be used.

Whenever possible, organizations should employ a comparison group against which the study group's progress can be compared. [Click here for an example of why comparison groups are important.] Simply looking at pre- and post-tests of a study group can be misleading for at least three reasons described below: selection effects, placebo effects, and climate effects.

For cost or other reasons, an evaluation using a simple pre- and post-survey is often the best organizations can do, but we wanted to alert you that such a test (without a comparison group) provides only a rough sense of the causal role of the program in social capital creation. Here are some other factors, besides the program itself that might explain the observed shift in social capital:

Selection Effects. First, the study group may be significantly predisposed to increase social capital or, alternatively, be an especially resistant population. These predispositions are critical because participants (and volunteers, and staff) are typically "self-selected"; that is, they themselves choose whether to get involved with a program. This selection bias means the intervention does not fit a standard experimental design model, in which individuals exposed to some treatment are randomly selected. These selection effects can bias the results positively or negatively.

In a positive selection bias example, individuals participate in a program because they are looking to change their lives ("getting their act together") and may be making other lifestyle shifts that also affect their social capital. In these cases, some or possibly most of the observed change in their social capital is because of participants' earlier decision to change their lives, not the result of the program.

In a negative selection bias example, people attracted to a program have below-average social capital; for example, community newcomers, people with certain disabilities, mental illness, or a substance abuse problem. In these cases, a program might have a very large impact because it is building off a very low base or a very small impact because of the difficulty of getting such a group to make connections with others. A proper assessment of the program would take these selection effects into account.

Placebo Effects. A second factor that skews the program's assessment of its causal role in social capital creation is the evaluation itself. This may sound strange, but by asking people about their social connections, memberships, and level of social trust, we stimulate them to think about these issues. When we ask them again a few months, even a year later, they may have given the questions more thought and have more accurate and less biased answers. The questions themselves may have prompted them to act to build their own social capital. This situation is somewhat analogous to the placebo effect that medical researchers sometimes find when patients assert improvement even though they were only given a sugar pill as part of a control group.

The Civic Climate. Finally, community changes occurring between the pre- and post-evaluation may distort our sense of program's impact on increasing social capital. We'll call this the "civic climate effect." Some of these shifts, such as a breaking political scandal, an economic downturn, or a local crisis, might temporarily depress the level of social trust in the general population. Some other changes, like a local sports team championship, might raise spirits and appear to raise social capital as well. If the pre- and post-tests are taken in different seasons, the climate effect may be just that, a result of the weather!

Because of factors like "selection," "placebo," and "climate" effects, the evaluation results are much more convincing if the change in social capital of the study group is contrasted with the corresponding shift in an appropriate comparison or control group. Because the addition of this group is an attempt to re-create a classic experimental design that includes "treatment" and "control" groups, this evaluation approach is often called quasi-experimental.

If a comparison group can be found that is similar to the study group in terms of their baseline level of social capital or, especially, their ability to acquire new social capital, then a comparison of the changes in social capital between the two groups will yield a much better estimate of the actual program impact. This can be a tall order. Some organizations will not have access to a good comparison group. For example, an organization that implements a school-wide initiative would ideally compare their success with a similar school where this initiative was not tried, but that requires access to another school. The evaluation of many workplace innovations has this problem. Comparisons are probably also not practical for those assessing community-wide impacts.

Finally, for those who deal with very specific populations, at-risk youth or single mothers for example, it is not always possible to identify an appropriate comparison group. By answering the question, "Who is comparable to this group?" we identify what we think are the relevant specific group characteristics for social capital. This is unavoidable theorizing, as with the program review in Step 3, but at least we can make these assumptions explicit.

Who are good comparison groups?
The simplest case involves comparisons of two program sites operated by the same organization, but where the program change is implemented at only one site. This is not a perfect comparison since one site might offer different programs, serve a different population, or have different staff. Still, the two sets of participants are often quite comparable.

Even better, for example, a school program with limited enrollment may admit qualified students by lottery, allowing one to compare admitted students with a random selection of those not selected in the lottery. These situations are very close to a true experiment and should produce good results. Sometimes this set-up is called a "natural experiment".

Many organizations don't have multiple sites, or may be unwilling to withhold a worthwhile program or service from a group of clients in the name of research. (This is a basic ethical problem that frequently arises in medical studies of interventions that are expected to help patients.) If an ideal control group is unavailable, what are some second best alternatives? Three possible sources for comparison groups should be considered:

(1) People affiliated with your organization but not in the program being evaluated. Some organizations may have access to such potential participants, for example kids who use a youth center for a basketball league, but not for the after-school mentoring program being evaluated. If all these kids come from the same neighborhood, the basketball league youth may be similar enough to use as a comparison group. It may be a good idea to draw a relatively large sample of the comparison group in case a subset of them emerges as particularly good (e.g., kids with single parents). Another example might be a community health center, evaluating a specific health program, recruiting other inpatients as a comparison group.

(2) Using a community organization or program as a comparison site, perhaps one interested in duplicating the program under review. For programs that serve children, partnerships with local schools might serve this purpose. For adult populations, it is a little harder to imagine where a comparison sample might be recruited. This option is obviously more plausible if community organizations and institutions have good working relationships.

(3) For programs serving the entire community, using the general population as a comparison group, i.e., contrasting the social capital of program participants with a sample of the entire community. The comparison group here can be drawn randomly from the listings in the appropriate telephone exchanges. If you wanted to further refine the sample you might filter or screen out candidates to make sure respondents are of the right age group, neighborhood, gender, homeowner status, or family structure (parents, single parents). For such a sample to be practical the desired characteristics must not be too rare (if they are, many more calls must be made and the cost or time to complete these surveys rises quickly).

Ongoing example: hear who Jumpahead used as comparison group ?

WHAT SHOULD WE MEASURE? (expand)

STEP 5: Conducting an Evaluation