Program Evaluation
Since its inception in the 2005/06 school year, more than 200 secondary schools have participated in Project P.A.T.H.S. Over the past few years, the research team has used different strategies to evaluate the effectiveness of the program systematically. The details are as follows:
In the Full Implementation Phase, 24 experimental schools and 24 control schools were randomly selected to participate in a randomized group trial in the 2006/07 school year. This trial was designed to investigate the longitudinal impact of the project. Using a longitudinal randomized controlled design, eight waves of data were collected over a period of five years (from the 2006/07 to 2010/2011 school year). Results demonstrated that participants in the experimental schools had significantly higher positive youth development levels than participants in the control schools at posttest, based on different indicators derived from the Chinese Positive Youth Development Scale. Adolescents who had participated in the program also exhibited significantly slower increases in delinquent behaviors and substance use compared to control participants. Moreover, participants in the experimental group who perceived the program as beneficial to their development had significant group effects on growth parameters compared to control participants.
Both students and program implementers were invited to complete subjective outcome evaluation forms (Form A and Form B, resp.) after completion of the program. These forms were designed to assess their perceptions of the program, the implementers, and the benefits of the program. Overall, the findings showed that different stakeholders had positive views about the program and that they perceived the program as beneficial to participants.
In the process evaluation, trained research assistants made systematic observations in randomly selected schools to assess the details of program implementation. The findings generally showed that the program implementation quality in these schools was high. In addition, several studies generally revealed that adherence to the program manuals (for activities and schedules) was high, with a mean adherence of over 80%.
To understand the process of implementation, interim evaluation was conducted by randomly interviewing roughly half of the participating schools in the Experimental Implementation Phase or the Full Implementation Phase. The findings generally showed that the participants and implementers perceived merits and benefits in the program, although they also observed difficulties in implementing the program and made recommendations for improving the program and the implementation process.
Student Focus Groups: focus groups with students from randomly selected participating schools were carried out. Results indicated that the program participants had positive impressions about the program and that they perceived the program as beneficial to them.
Focus Groups with Program Implementers: focus groups with program implementers in randomly selected participating schools were also conducted in the project. Generally speaking, the program implementers had positive evaluations of the program. They perceived the program as beneficial to students. They also gave suggestions for how the program could be improved.
In-depth Interviews with Program Implementers: in-depth interviews were carried out to gain an in-depth understanding of implementers’ perceptions of the program. Results showed that the program implementers had positive evaluations of the program and that they perceived the program as beneficial to the students.
Case Studies: various case studies were carried out in different schools. Results showed that the Tier 1 Program was successfully incorporated into the school curriculum. Moreover, the program implementation facilitated factors that were closely related to the “5 P’s” model (i.e., program, people, process, policy, and place).
Evaluation Based on Student Weekly Diaries: after completion of the Tier 1 Program, students from the participating schools were randomly selected to write a reflection journal in the form of weekly diary. These diaries were designed to assess their perceptions of and feelings toward the Tier 1 Program and the related benefits. Results showed that most of the respondents had positive views toward the program, had positive views toward the instructors, and stated that they had acquired competencies at the societal, familial, interpersonal, and personal levels after joining the program. They also saw the program as beneficial to them.
Information Collected in the Co-Walker Scheme: analysis of anonymous data showed that the implementation of the project in the schools was generally positive and smooth.
At the end of the Full Implementation Phase, students were randomly selected to complete repertory grid tests. These tests were used to assess how the participants perceived changes in their identity at different points of time. The findings generally showed that the participants had improved self-representations after joining the program.
The participants were invited to complete subjective outcome evaluation forms (Form C) after completion of the Tier 2 Program. These forms were designed to assess their perceptions of the program, the implementers, and benefits of the program. Results showed that participants had positive views toward the program, the implementers, and the benefits of the program.
The existing evaluation results have consistently shown that different stakeholders had positive perceptions of the project. They perceived that the Tier 1 and Tier 2 Programs could facilitate positive development in participants. The respective evaluation results have been published in special issues of six international journals (Shek, 2008; Shek & Ma, 2008; Shek, Ma, & Merrick, 2008; Shek & Merrick, 2009; Shek & Merrick, 2009; Shek & Ma, 2010) and in several articles. The following articles summarize all of the evaluation results of the project over the years. For all our published teaching materials, books, and articles, please refer to the Published Literature page on this website.
Successfully submitted.