Since its inception in the 2005/06 school year, more than 200 secondary schools have participated in Project P.A.T.H.S. Over the past few years, the research team has used different strategies to evaluate the effectiveness of the program systematically. The details are as follows:
Objective Outcome Evaluation
In the Full Implementation Phase, 24 experimental schools and 24 control schools were randomly selected to participate in a randomized group trial in the 2006/07 school year. This trial was designed to investigate the longitudinal impact of the project. Using a longitudinal randomized controlled design, eight waves of data were collected over a period of five years (from the 2006/07 to 2010/2011 school year). Results demonstrated that participants in the experimental schools had significantly higher positive youth development levels than participants in the control schools at posttest, based on different indicators derived from the Chinese Positive Youth Development Scale. Adolescents who had participated in the program also exhibited significantly slower increases in delinquent behaviors and substance use compared to control participants. Moreover, participants in the experimental group who perceived the program as beneficial to their development had significant group effects on growth parameters compared to control participants.
Subjective Outcome Evaluation (Tier 1 Program)
Both students and program implementers were invited to complete subjective outcome evaluation forms (Form A and Form B, resp.) after completion of the program. These forms were designed to assess their perceptions of the program, the implementers, and the benefits of the program. Overall, the findings showed that different stakeholders had positive views about the program and that they perceived the program as beneficial to participants.
In the process evaluation, trained research assistants made systematic observations in randomly selected schools to assess the details of program implementation. The findings generally showed that the program implementation quality in these schools was high. In addition, several studies generally revealed that adherence to the program manuals (for activities and schedules) was high, with a mean adherence of over 80%.
To understand the process of implementation, interim evaluation was conducted by randomly interviewing roughly half of the participating schools in the Experimental Implementation Phase or the Full Implementation Phase. The findings generally showed that the participants and implementers perceived merits and benefits in the program, although they also observed difficulties in implementing the program and made recommendations for improving the program and the implementation process.
Student Focus Groups: focus groups with students from randomly selected participating schools were carried out. Results indicated that the program participants had positive impressions about the program and that they perceived the program as beneficial to them.
Focus Groups with Program Implementers: focus groups with program implementers in randomly selected participating schools were also conducted in the project. Generally speaking, the program implementers had positive evaluations of the program. They perceived the program as beneficial to students. They also gave suggestions for how the program could be improved.
In-depth Interviews with Program Implementers: in-depth interviews were carried out to gain an in-depth understanding of implementers' perceptions of the program. Results showed that the program implementers had positive evaluations of the program and that they perceived the program as beneficial to the students.
Case Studies: various case studies were carried out in different schools. Results showed that the Tier 1 Program was successfully incorporated into the school curriculum. Moreover, the program implementation facilitated factors that were closely related to the "5 P's" model (i.e., program, people, process, policy, and place).
Evaluation Based on Student Weekly Diaries: after completion of the Tier 1 Program, students from the participating schools were randomly selected to write a reflection journal in the form of weekly diary. These diaries were designed to assess their perceptions of and feelings toward the Tier 1 Program and the related benefits. Results showed that most of the respondents had positive views toward the program, had positive views toward the instructors, and stated that they had acquired competencies at the societal, familial, interpersonal, and personal levels after joining the program. They also saw the program as beneficial to them.
Information Collected in the Co-Walker Scheme: analysis of anonymous data showed that the implementation of the project in the schools was generally positive and smooth.
Evaluation Based on Repertory Grid Test
At the end of the Full Implementation Phase, students were randomly selected to complete repertory grid tests. These tests were used to assess how the participants perceived changes in their identity at different points of time. The findings generally showed that the participants had improved self-representations after joining the program.
Subjective Outcome Evaluation (Tier 2 Program)
The participants were invited to complete subjective outcome evaluation forms (Form C) after completion of the Tier 2 Program. These forms were designed to assess their perceptions of the program, the implementers, and benefits of the program. Results showed that participants had positive views toward the program, the implementers, and the benefits of the program.
The existing evaluation results have consistently shown that different stakeholders had positive perceptions of the project. They perceived that the Tier 1 and Tier 2 Programs could facilitate positive development in participants. The respective evaluation results have been published in special issues of six international journals (Shek, 2008; Shek & Ma, 2008; Shek, Ma, & Merrick, 2008; Shek & Merrick, 2009; Shek & Merrick, 2009; Shek & Ma, 2010) and in several articles. The following articles summarize all of the evaluation results of the project over the years. For all our published teaching materials, books, and articles, please refer to the page on this website.
This Evaluation Manual is designed for workers who implement the Tier 1 Program and Tier 2 Program of the P.A.T.H.S. Project. It is applicable to the programs implemented from the Secondary 1 to Secondary 3 levels.
This evaluation manual is designed to help workers evaluate the Tier 1 Program and Tier 2 Programs of the P.A.T.H.S. Project. Because evidence-based practice is emphasized in different disciplines, one unique characteristic of this project is to build in systematic program evaluation mechanisms. Although different evaluation mechanisms have been devised to evaluate the effectiveness of the P.A.T.H.S. Project (including randomized group trials, process evaluations, and qualitative evaluations), workers implementing the Tier 1 and Tier 2 Programs are required to conduct subjective outcome evaluation. In fact, it is our belief that evaluation should be a part of common practice for teachers, social workers, and other allied professionals, and it is not an extra research burden unrelated to the service delivered.
In case of download problems, please contact the Training Team at 3400-8510.