Kennedy School Saguaro Seminar

PROGRAM EVALUATION GUIDE
Phase One: "PLANNING"

OVERVIEW

STEP 1: Mobilizing Resources

STEP 2: Understanding Social Capital as it Relates to Organizational Mission

STEP 3: Identifying Program Links to Social Capital

Once the organization, in its social capital retreat has restated its mission in social capital terms, it can move on to an examination of the specific program or programs to be evaluated. Staff and volunteers who work directly with participants and the public must be very involved in this step. We recommend that you bring a diverse group of stakeholders together for your social capital retreat. While this diversity of thought may make it more difficult to conduct such retreats and manage the conversation, it increases the chance that you will not miss out on an important observation. The ultimate decision as to what you want to evaluate is yours, and this may be important to identify up-front, but you will benefit from diverse perspectives.

We expect that program specifics will be discussed in a second meeting that may be held a week or more after the first meeting or meetings that dealt with the issue from an organization-wide perspective. If the members of the organization cannot answer some of the questions that follow, it may be necessary to undertake a small internal "study" of program operations.

Note: If your organization operates only one program, the thinking just done in Step 2 may significantly overlap with this step.

PROGRAM'S RELATION TO COMMUNITIES SERVED (expand)

SOCIAL CAPITAL FORMATION STRATEGIES (expand)

WHICH TIES ARE ORGANIZATION TRYING TO BUILD? (expand)

IS SOCIAL CAPITAL AN ESSENTIAL PART OF PROGRAM OPERATIONS? (collapse)

    Whether the social capital identified in the above inventory of program-related ties is an essential element or by-product of a program will affect the evaluation design, so this should be discussed.

    1) WHAT SOCIAL CAPITAL IS ESSENTIAL FOR PROGRAM EFFECTIVENESS?

    The success or failure of some programs hinges directly upon social capital. There are two ways in which this occurs.

  • Key program goals may be expressible in social capital terms, so that program success is virtually synonymous with social capital creation. For example, programs expressly seeking to "build community" embrace social capital building as a primary goal, regardless of how they seek to effect this goal. It is certainly appropriate to evaluate these programs in terms of their success at building those features of community that they have explicitly targeted.

  • Social capital within the program may be essential to operate effectively. For example, a school-based program may believe that cooperative and trusting relationships between teachers and students are essential for the program. Note that members of the organization should identify and agree upon these ties in theory because they reflect how the program ought to work, although further reflection may reveal that there is little or nothing about the program that is actually likely in practice to build such ties.

    2) IS SOCIAL CAPITAL A BY-PRODUCT OF SOCIAL OPERATIONS?

    If increasing social capital seems like a frill rather than a key determinant of program success, it is a by-product. This determination could be the source of some discussion and disagreement. For example, some might view establishing a coffee hour after meetings as a key way to deepen relationships among program staff and participants. Others might view this as a wasteful use of space, cleanup time, time away from pressing work, and resources. Similarly, one daycare provider might see opportunities to create interactions among parents and staff as tangential to the program effort, another may see this as the best way that the daycare center can serve these families.

    Do not gloss over this step. Because members of the organization and the participants in such social capital retreats know their programs and organization well, it is tempting to give this step short shrift in the evaluation process and move on to concrete design questions. But evaluation design depends on specifying the pathways of social capital creation at this step, so we advise you to move slowly and be very thorough. We are interested at this point in all the interactions that might be important, not just the principal one(s).

    To return to our educational program example, the program might be viewed as only involving classroom interactions, when non-classroom interactions might be far more important setting for social capital creation. These social ties might range from student-teacher interactions outside traditional instructional settings, student-student interactions, and parent-student or parent-teacher interactions. Based on their programmatic expertise, members of the organization must decide which ties are central to program effectiveness. We hope that this guide and the prior discussion of social capital will help you decide this.

    Some program evaluation occurs even at this theoretical step. Fundamentally, interactions which programs are striving for on paper must include a plausible pathway by which the program forms these. If parent-teacher interactions are vital in theory, does the program's design create reasonable opportunities for these interactions? How are program participants supposed to interact with each other and with the staff and volunteers who run the program? While it will be difficult pragmatically to separate out the social capital effects of each type of interaction, an analysis of program goals and structures should at least suggest which of these relationships ought to be important. It may be helpful to draw a theory model that traces out how you expect to have the impact. For an example of this, click here [PDF].

    Ongoing example: check in with Jumpahead

Go on to Phase Two

 

 

TABLE OF CONTENTS

INTRODUCTION

PHASE ONE | Planning

  • Step 1: Mobilizing Resources
  • Step 2: Understanding Social Capital as it Relates to Organizational Mission
  • Step 3: Identifying Program Links to Social Capital

PHASE TWO | Evaluation

  • Step 4: Designing the Evaluation
  • Step 5: Conducting an Evaluation

PHASE THREE | Action

  • Step 6: Interpreting the Results
  • Step 7: Revising Programs

GLOSSARY


This guide was created by
Thomas Sander, Executive Director of the Saguaro Seminar, &
Stephen Minicucci, Ph.D.,
Principal Investigator

Edited and adapted for the web by Benjamin Toff

E-mail us your ideas for improving this Guide.


SEARCH THE SAGUARO WEBSITE
home | about Social Capital| contact us | email us | KSG | Harvard | site map

Copyright, 2007, President and Fellows of Harvard College

website design by WhiteDogDesign.com

go to main harvard site