|
|
 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
PROGRAM EVALUATION GUIDE
Following along with the example of the fictional non-profit called "Jumpahead" will help illustrate the various stages of the evaluation process.
THUMBNAIL SKETCH
Description of program: Jumpahead pairs highly trained college students with preschool children struggling in Head Start programs, and through one-on-one tutoring improves school readiness. The program generally placed tutors (usually 1 or 2) in a classroom as a teacher's aide, and the tutors tutor Headstart youth (tutees). There are some important group activities among all the Jumpahead tutors, but only approximately once a week.
PLANNING - Step 1: Mobilizing Resources
Jumpahead had made a commitment to their funder (as a precondition of getting any more funding) that they would show whether the meetings of Jumpahead tutors with the tutees' parents worked. Jumpahead didn't have lots of resources, but they trained some volunteers from a local university to call up parents. They decided that they would get some other undergraduate volunteers who had good quantitative skills to enter the results of the survey in an Excel spreadsheet and to tabulate the results.
PLANNING - Step 2: Understanding Social Capital and Your Organization
First social capital retreat: Jumpahead held a two hour meeting to consider how social capital related to their mission and what types of social ties they were creating. They invited the organization's top staff of the organization, two board members, and a couple of Jumpahead tutors.
WHAT TYPES OF SOCIAL CAPITAL TIES YOU ARE BUILDING?
Jumpahead's hunch of the inventory of what types of social ties they were creating was as follows:
- Jumpahead was building bridging social capital since tutors and tutees (and their parents) came from very different socio-economic swathes.
Is Jumpahead trying to increase bonding or bridging social capital?
Jumpahead was trying to build some of each but especially build bridging social capital between the tutors and the tutees' parents (which generally came from different races and classes than the tutors). (Go back) WHAT ARE YOUR ORGANIZATIONAL GOALS?
Do you believe in building more social capital because it is an essential strategy of how to provide your services or is it an ancillary benefit of your program, but not its key goal?
Participants in Jumpahead's retreat didn't believe that social capital was the critical thing for the organization, but they thought that social capital was pretty important for the organization to be effective. Hunches were as follows:
HOW DOES ORGANIZATIONAL MISSION MAP ONTO SOCIAL CAPITAL? Jumpahead's mission focused on increasing school readiness among HeadStart children through tutoring. They sensed that the quality of the personal relationships between the Jumpahead tutors and the HeadStart tutees and the relationship of the Jumpahead tutors to the tutees' parents were critical.
Which of the four social capital goals is predominant for your organization?
Jumpahead was trying to increase Individual Bonding and Bridging. (Go back)
WHAT COMMUNITY ARE YOU SERVING? Jumpahead basically served a program community since the tutees were all defined by who was in which HeadStart programs. While it was true that all tutees were young and poor and that most tutees were of color, their eligibility to get tutored by Jumpahead tutors was determined by their attending partner HeadStart centers. (Go back)
PLANNING - Step 3: Identifying Program Links To Social Capital
Who attended second social capital meeting: To the second meeting (a 2.5 hour meeting) Jumpahead invited some of their senior staff, some prior Jumpahead tutors and some prior parents (so as to not influence the folks they were going to monitor), funders, some community partners (i.e., HeadStart centers). [Jumpahead was worried that involving some current parents would tip the objectivity of responses when those parents were later surveyed.] At this meeting they discussed their theory of how the specific program that they were looking at was likely to build social capital.
What the inventory of social ties looked like: Jumpahead only ran one program so their inventory of social ties for their program looked the same as the inventory for their organization. To see their inventory for their organization (from step 1), click here.
PROGRAM'S RELATION TO COMMUNITIES SERVED
Jumpahead, as with many programs being evaluated, had the aim of heavily serving program participants (their tutees). (Go back)
SOCIAL CAPITAL FORMATION STRATEGIES
Jumpahead primarily expected to build social capital directly by fostering social ties: for example, between the parents of tutees and the tutors through their pizza get-togethers, between the tutors and tutees through the tutoring, among the tutors in their weekly get-togethers, etc. (Go back)
WHICH TIES ARE ORGANIZATION TRYING TO BUILD?
Jumpahead was primarily trying to link program staff and volunteers (the tutors) to their participants and their families (the first of the six classes of ties mentioned). (Go back)
IS SOCIAL CAPITAL AN ESSENTIAL PART OF PROGRAM OPERATIONS?
Jumpahead definitely thought social capital was far more important than a by-product. Although most sensed that it would be much harder for the organization to achieve its goals without social capital, there was no consensus that the organization couldn't achieve its goals without social capital. (Go back)
EVALUATION - Step 4: Designing the Evaluation
Jumpahead decided to measure the relationships between relationships between Jumpahead college students (tutors) and the parents of the children-tutees in the Head Start programs where Jumpahead tutors were engaged.
They were concerned that unless they got the families more engaged in their kids education, they would not succeed and they wanted to see whether the social capital had increased.
HOW CAN WE BEST IMPLEMENT A BASIC "BEFORE AND AFTER" STUDY FRAMEWORK?
When they decided to do pre- and post-surveys?
Jumpahead conducted the pre-survey within the first week of the program (prior to when the tutors would have had their first meeting with tutees). They decided to conduct their post-survey 5 weeks after graduation, since the organization was concerned that there would be too much program euphoria right around graduation that might distort the results. (Go back)
SHOULD WE USE A COMPARISON GROUP? AND, IF SO, WHO?
Jumpahead had just started a program in a second community, but thought it was too fledgling and its staffing too tenuous to be a good control group. They decided to enlist another program (KidHelp) that tutored kids in Head Start in the same town as a control group and had the KidHelp program agree to have the parents of all the 70 KidHelp tutees be interviewed. There was one problem with this control group. Since KidHelp didn't actually work at all with the parents, it was hard to get their contact information and get them to agree to be called. The response rate among KidHelp parents, i.e., the percent of parents contacted who agreed to be surveyed, was as a result lower than for Jumpahead parents, but there did not appear to be a non-response bias. (Go back)
Jumpahead decided to ask all the recommended core social capital questions and all the recommended demographic questions. In addition, here are some of the program-specific questions that Jumpahead asked the parents. [These questions were all asked only in the POST survey, except for question #7 and #8]
1. on how many pizza get-togethers did you go to with the tutor
1a. if you went to fewer than 6 [the number that had been intended], what was the reason
2. how many additional times (other than pizza dinners) did you see your child's tutor in person over the past 12 months
2a. other than scheduling get-togethers, how many times did you call your child's tutor or did your tutor call you in the last 12 months.
3. How much do you think that the Jumpahead tutor can be trusted
4. how much do you like your child's tutor?
5. [instrumental questions about the social tie with the tutor] Could you go to your child's tutor if you...
6. how helpful do you think your child's tutoring in HeadStart was?
7. how often do you read to your child? [ASKED PRE-SURVEY AND POST-SURVEY]
8. how often do you try to reinforce anything that your child is learning at HeadStart? [ASKED PRE-SURVEY AND POST-SURVEY]
9. what did you talk about during your get-togethers (choose all applicable)
10. We find with many parents of children in HeadStart, that their parents never learned how to read. Could you tell me what your proficiency in reading is?
11. Do you live with any other adult (a relative, a husband or wife, a close friend)?
12. If yes, is there someone in the house who can read easily?
(Go back)
EVALUATION - Step 5: Conducting the evaluation
Jumpahead decided to do phone surveys, but sent parents a note in advance letting them know that an individual would call them, that responses would be anonymous and that Jumpahead would really appreciate their candid responses. [Jumpahead decided not to conduct in-person interviews of parents because the parents were not regularly in Jumpahead centers and Jumpahead thought it would be too time-consuming to have the volunteers visit the parents at their homes. Also, Jumpahead believed they would get more candid responses in the phone survey interview format.] (Go back)
Since Jumpahead only had 50 kids in their program, and there were only 70 parents of these kids (since many children had "single moms"), they decided to measure the levels of social capital among all 70 parents, rather than choose a sample. [If they had been involved with e00 kids, for example, they certainly would have had to choose a representative sample.]
MAKING SURE SAMPLES ARE RANDOM (AND NON-RESPONSE ERROR)
Jumpahead didn't need to worry about random samples since they were conducting a census sample of all the children in their Jumpahead program.
WHO SHOULD CONDUCT THE EVALUATIOND?
As mentioned earlier, Jumpahead had the surveys conducted by trained volunteers. (Go back)
ACTION - Step 6: Interpreting the Results
Using the approach taken in the "Data Analysis" section, Jumpahead computed the following percentage increases or (decreases) in the questions they asked. [Note: this is not nearly a complete list of all questions asked.]
Note: increases/decreases measure the increase or decrease in the frequency of responses between the baseline survey and the post-survey. For example if respondents socialized with co-workers 20 times a year in the baseline survey and 22 times a year on average in the post-survey, that is an increase of 10%. The third column compares the percentage change in the study group (Jumpahead) with the percentage change in the control group (KidHelp).
Using the indices created from the Social Capital Community Benchmark Survey, these were the results that Jumpahead got on a few social capital indices.
Jumpahead tried to see what factors might be responsible for the increases, and looked at how involvement with pizza get-togethers affected the results. Although the sample sizes started to get quite small (and hence their conclusions were more sketchy), they found (by combining the survey with reading scores that the HeadStart centers provided at the start and end of the academic year) that:
ACTION - Step 7: Revising Programs
Jumpahead used the data in the following ways.
First, they reported the findings to the funder who was interested in enabling this program to spread elsewhere, and wrote up their findings in a newsletter to other HeadStart centers to encourage them to find ways (like Jumpahead) to get parents more involved.
Second, Jumpahead decided to make this program (pizza get-togethers) a mandatory part of all programs (at their new site), but they made the following changes:
1) they encouraged tutors to spend the first meeting or two just getting to know the parents and not talking about how to reinforce the tutee's learning (unless the parent raised it), so as to not put off the parents;
2) Jumpahead encouraged tutors in the third meeting to tell parents that parents who knew tutors better and knew what their child was learning about improved much more than others; and
3) Jumpahead tutors were sensitized to ask about reading levels in the family. Jumpahead tutors explained that literacy was lacking in many of these families, and if there were literacy problems among the parents, Jumpahead tutors made far greater efforts to try to get the child's parents into literacy programs.
4) For Jumpahead tutors who hadn't held two meetings with their tutees' parents by November 30 of each year, a Jumpahead staff person followed up with the parents to reiterate how important these meetings were and to see if there were any issues and anything that Jumpahead could do to make these meetings happen.
|
TABLE OF CONTENTS PHASE ONE | Planning
PHASE TWO | Evaluation PHASE THREE | Action This guide was created by |
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|
|
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||
