SAGUARO SEMINAR - Civic Engagement in America

Data entry: The first step of interpreting the results after you have conducted the survey is to enter the results of the survey in a database or a software package. Excel is fine if you simply want to know what the averages were on the various questions or other simple statistical functions (like standard deviation, etc.). If you want to do more sophisticated analysis, like for example correlate the level of volunteering with the intensity of program involvement, and control for the participant's education level, you will probably need a statistical package (like SAS, SPSS, Stata, etc.).

In general, each row of your database will correspond to a different survey respondent and will probably have a field indicating the respondent ID number (to preserve confidentiality), and each column will correspond to a different question asked (like the gender of the respondent, or how many times they volunteered in the last 12 months, or how many social events of the program they attended, etc.). In statistical packages each cell would have a coded number for that answer and you could then see, for example, that a "2" under the column "RELIMP" meant that religion was somewhat important to the respondent and a "1" meant it was very important.

Anyone (a volunteer, a program participant, etc.) can enter the data as long as he/she didn't know the identity of the respondent. But it is critical that the responses be entered accurately as it will produce erroneous analysis later if the answers are misentered.

Data analysis: For each question, determine the percentage change among the study group between the pre- and post-survey, and compare it to analogous percentage change among the control group. To see how to do this, click here.

You may want to think about combining some of the social capital questions into indices along some of the 10 dimensions (like social trust, inter-racial trust, etc.). You may want to determine some overall social capital indices based on the general social capital questions or based on the program-specific questions. [See "Determining Data Weighting" below.]

If you are well versed in data analysis, or have an evaluator or local college or university helping you, you may want to conduct more sophisticated analyses to try to understand what is driving some of these changes. For example, are the changes between pre- and post-survey responses not true across-the board and only for respondents of a particular gender, age range, or marital status? Are the factors driving a decrease in respondents' social trust different from the factors leading to an increase in respondents' social networks? Etc.

Determining data weighting: Prior to your analysis and reporting on your results, you should figure out whether you intend to summarize the data into certain indices showing program impact. This is something that you may want to consider back at Step 5 ("Constructing the questionnaire"); your results will have more integrity if you form this weighting prior to looking at the data, rather than first entering the data and then seeing which weightings of the data and variables show a significant improvement or prove your hypotheses. For example, you might decide that you want to combine 10 questions and have them each count equally. Conversely, maybe 20 questions are important, but 3 of them (for example on bridging social capital where the program put its emphasis) should count much more than others.

Report on your analysis: If you are sharing the results of your evaluation with an external audience (funders, policy makers, the surrounding community, etc.) and/or quite possibly if you are sharing them with program staff, you will need to write up your findings in a report. [For a very brief checklist of some things to consider in writing a report, click here.] If you are sharing this with external audiences, you probably want to write a draft version of this report, circulate it to program staff and get their reaction, before revising it and sharing it externally. You'll have to decide how much of the findings you are willing to share with the external audience, although generally it is less ethical to only report on those items showing social capital improvement and not to report items showing no change or even negative social capital impact.

Assuming you decide to do a report, it should probably contain some or all of these sections:

Executive summary/Intro: A quick summary of why you undertook the evaluation, how you measured the impact, and your headline findings.
Program description: a thumbnail description of what your program does, the mission of your organization and the population it serves.
Goals and objectives of study: why you undertook the study, and what specifically you were attempting to measure (for example, whether children in a school who were randomly assigned to participate in a mandatory service learning program showed higher trust and stronger friendship networks than ones who didn't).
Research design: how you asked the questions, who you surveyed, etc.
- Data collection procedures/Methodology: how you went about gathering the data (for example were the surveys done by phone, in person, mailed, or self-administered), where did you survey individuals; when; who collected the data (an outside firm, trained volunteers), how many people were surveyed, whether and how confidentiality was maintained; who entered the survey; any incentives given to respondents to participate; how many follow-up calls were made for hard-to-reach individuals; etc.
- Instrument: what questions you asked of your study and control groups
- Sampling plan: how you selected individuals for your study and control groups (assuming that you didn't survey everyone in that group)
Data analysis/results: what were the more interesting findings (good and bad) from the data, and in the report (or an appendix) the change you observed on all of the questions asked (showing means, and possibly standard deviations for all questions pre- and post-survey, and with the corresponding pre- and post- means for the control group if you had one)
Implications/ recommendations for programmatic change: (see discussion immediately following)

The final stage in program evaluation is making recommendations, which can be politically charged, and is one advantage of using external researchers. We recommend that a lot of dialogue take place between the research team, agency staff, and, if possible, the program participants, in drafting the recommendations. You should consider whether a potential new program element ought to be expanded (if it is showing the impact you expected), retrofitted (if it is not showing the impact you hoped for in places or at all), or possibly discontinued.

If the data show findings different from what you expected (for example, you observed expected large increase in volunteering, but unexpectedly found no increase in bridging social ties), you might think about whether there are things that your program could be doing to make it easier to forge such relationships. Some sample questions in such a case that you might consider are:

Is this an issue about the composition of your program participants (or staff) or the types of partners with whom you work?
Would having smaller diverse groups as part of your program facilitate this?
Are there social activities that you could foster that would make it easier for these ties to form?

Depending on your survey, this process of thinking about program recommendations may involve multiple steps. Looking at and discussing the data may raise additional questions that you may be able to go back to the survey data to test. If, for example, you are not observing a big increase in political participation among participants, this may raise additional questions. Does it depend on how many program sessions the participant attended? Does it depend on whether the participant had a mentor in the program? Did women and men or rich or poor participants show different trajectories? All of these questions can help tease out what is going on in a program and suggest why things might or might not be working as you thought.