Once an organization has identified the social capital elements in its programs and specified pathways through which its activities contribute to social capital formation, it can formulate plans to evaluate its social capital impact. Somewhat artificially, we break down this process into steps 4 and 5: Step 4 involves the basic design questions, such as the study's timeframe and the choice of study populations; Step 5 concerns the nuts and bolts of fielding a survey-based evaluation. You can also access a list of social capital survey questions intended to help organizations develop their own survey instruments.

STEP 4: Designing the Evaluation

The design of the evaluation hinges on answering a sequence of questions discussed below. The answers to these questions flow directly from the earlier discussion of what the program is trying to do.

WHO IS THE STUDY GROUP? (expand)

HOW CAN WE BEST IMPLEMENT A BASIC "BEFORE AND AFTER" STUDY FRAMEWORK? (collapse)

Once the decision about whom to study is made, the next basic design question concerns the type of comparison upon which the organization will base its evaluation.

In all cases, we expect organizations to conduct before and after surveys of the social ties and social capital. Basically, you interview the study group (for example, program participants) and ask them questions before exposure to the program (for example, early on the first day). This "before" survey is also called the baseline. Then, you question them "after" a program cycle,¹ which, depending on how long the program lasts, might be a semester later, a year later, etc.² The increase or decrease of specific questions is a reasonable estimate of how much of that type of social capital was created or lost over the program cycle. However, a number of circumstances complicate this simple design.

The model is difficult to apply in cases where participation is spontaneous and the program cycle very short - for example, one-time community events and other programs open to the general public. In these cases, a short baseline survey can be administered during a community event, or immediately afterwards. It is not practical to gather baseline measures of attitudes (such as trust in others or attachment to the community) since one typically cannot talk with participants before they are exposed to the one-time event (e.g., a festival). However, answers to actual behavioral questions (such as the level of recent social activities or voluntarism) may form more valid baseline measurements.³ You might try to obtain e-mails or phone numbers and follow up with these interviewees in a month or a year to see if there seems to be any persistence to the social capital responses of the participants.

The model is also difficult to apply in programs that mix new and continuing participants. We assume that the study group will be a sample of new participants. Note that a simple comparison of the social capital of new and longtime participants, while possibly very useful, will not represent a good test of social capital creation because participants choosing to return to the program may differ systematically from the truly new participants. For example, participants might get the largest social capital benefit from their first year of program involvement. Including many returning participants would thus dampen the program impact demonstrated because many of those sampled would show smaller social capital gains in their past year than first year participants. Likewise, if you are interested in the social capital among returning participants, including the responses of new clients may alter the true results for this group.

A less serious problem with mixing new and returning participants is that it forces comparison between groups of participants while the before-and-after design employs comparisons of individual participants. While both comparisons are valid, comparisons of individuals (rather than groups) are more likely to produce statistically significant results.

Task 2: Determine when to conduct the baseline or "before" measurement.

There is not a lot to be said about this task. You should ideally conduct the baseline measurement among the study group at or just before the start of the program, if this is possible. Elsewhere we discuss how many people need to be surveyed, survey question design, and making sure that you chose a representative population (if you are not surveying everyone). If you employ a comparison group (discussed in the next section), you should conduct the "before" measurement of this comparison group at the same time as the study group (if possible).

Task 3: Determine when to conduct the "after" measurement.

You must also consider when to conduct the post-survey of this pre- and post-measurement. The answer will depend on the types of impacts that are desired and on practical issues of conducting interviews.

For many programs, the best option is an "exit interview" in which the interviewer questions the participant on site near the end of his/her program exposure (or at the end of a cycle of a continuing program). "Entrance" and "exit" interviews then form the basis of a comparison. The strength of exit interviews is that respondents are generally willing to be interviewed and answering questions is easier since the program details are still fresh in their minds. If participants are likely to continue their relationship with the program (or with the organization), exit interviews are particularly appropriate. There are two cautions however:

(1) Difficulty in discerning longer-term impacts: Some anticipated program effects, such as becoming involved in other community organizations, or in local politics, may take much longer to emerge and therefore not be evident in such an early "after" snapshot. If the evaluation team chooses the exit interview approach, they should de-emphasize asking about social capital effects not expected to emerge in the short-run.

(2) Social Capital Euphoria: The exit interview may exaggerate other social capital aspects, such as participant-to-participant ties: many respondents may report making long-term friends through the program that they will not in fact maintain. Also, depending on the interviewer, participants may feel pressure to give politically correct answers; for example, claiming that they do more volunteering if the organization with which they have been involved has an ethos of volunteering.

An alternative to an "exit" interview is conducting "follow-up" interviews some time after program exposure. The strength of this approach is that it reduces these problems-long-term social capital building may be more evident and short-term "euphoria" may have abated. The key weakness with the follow-up approach is that respondents are harder to locate and schedule for interviews and they are more likely to refuse interviews. Also, the further into the future that the follow-up survey is done, the harder it will be for the organization to show that they were responsible for the result rather than other things that participants had done since. As a practical matter, the follow-up interviews are more likely to be conducted over the telephone (see discussion in Step 5 below). When should such follow-up interviews take place? Two to six weeks will likely ensure that the short-term program euphoria has passed, but six months or a year are likely necessary for real long-term effects to become evident. However, you should note that response rates will continue to decline the longer into the future you wait.

Persistence: A Third Survey? For those cases in which long-term effects are the central concern, evaluators should consider a three-step evaluation in which clients are interviewed on entry, exit, and then some time later (perhaps a year), assessing different aspects of social capital in each snapshot.

Ongoing example: hear what group Jumpahead decided to evaluate and what social ties they decided to measure ?

1. One should conduct the survey at the end of the same program cycle for all participants regardless of whether they intend to do another program cycle.
2. For programs with no fixed program cycle, e.g., a homeless shelter with no limit on how long residents can stay there, find a common denominator of how long people stay with the program, or if there is not such commonality set the follow-up survey artificially at a year later.
3. While in principle program exposure should not affect questions of a respondent's past behaviors, the problem is that many social capital questions have socially desirable responses (e.g., people would like to say that they recently volunteered), which elements of your program elements may augment.

SHOULD WE USE A COMPARISON GROUP? AND, IF SO, WHO? (expand)

WHAT SHOULD WE MEASURE? (expand)

STEP 5: Conducting an Evaluation