______________________________

artsJAM Detroit!

Work Alternatives for Youth

          _____________________________________________________

 Program Evaluation Summary

                                   by Michael Gielniak, Lead Researcher

                                                               with Philippe Bitton, Research Assistant

______________________________

I. INTRODUCTION

Background

The primary goal of the artsJAM Detroit! Work Alternatives for Youth program is to provide arts education for youth (ages 14-26) with disabilities to increase their artistic skills and prepare them for the world of work. The goal of this study, however, is to provide VSA arts of Michigan an evaluation of the program's ability to develop the skills necessary to help students with disabilities transition to the world of work.

VSA arts of Michigan partnered with the Skillman Foundation, the Youth Sports and Recreation Commission of the cities of Detroit, Highland Park and Hamtramck, special education administrators in the Detroit Public Schools, Highland Park Community Schools and Hamtramck Public Schools, and the VSA arts national organization in Washington D.C., to provide funding for the program.

Participants

An application process was conducted for the artsJAM Detroit! Work Alternatives for Youth program toward the end of the 2003-2004 school year. Well over 100 students from Detroit, Hamtramck and Highland Park applied for the 25 positions in the program. School counselors and social workers were asked to recommend students for the program, and then VSA staff conducted interviews and selected participants. Students participating in the program are emotionally, cognitively, and physically impaired. Three participants are autistic. Criteria for selection included:

·          Students receiving special education services in their school district

·          Previous participation in the artsJAM Detroit! program

·          Representation from a variety of individual schools within the three

participating school districts

Basic demographic information about participating schools is listed below.

Detroit Public Schools

3031 West Grand Blvd, Suite 485
Detroit, Michigan 48202

Detroit Publics Schools is a large urban district that faces the typical challenges of most urban public schools. The district services over 145,000 students of which almost 70% are economically disadvantaged.  The passing rate for the district on the state standardized test is 30%, which is far below the state average. The percentage of students receiving special education services is 13%, which is above the state average*.

               Table 1

                Table 2

Hamtramck Schools

3201 Roosevelt

Hamtramck, Mi 48212

Hamtramck Schools is a small district within the boundaries of the city of Detroit. The district faces tremendous challenges because of their highly diverse population of non-English speakers. The district services 3,657 students of which over 80% are economically disadvantaged.  The passing rate for the district on the state standardized test is 25%, which is far below the state average. The percentage of students receiving special education services is 6.5%, which is below average in the state*.

               Table 3

                Table 4

Highland Park Community Schools

20 Bartlett Street

Highland Park, Mi 48203

Highland Park Community Schools is also a small district within the boundaries of the city of Detroit. Although it is not a large urban district, Highland Park faces the same challenges as other urban districts dealing with high levels of poverty. The district services 2,968 students of which approximately 67% are economically disadvantaged.  The passing rate for the district on the state standardized test is 15.5%, which is among the lowest in the state. The percentage of students receiving special education services is 14.6%, which is above the state average*.

               Table 5

                Table 6

Methods

Tools

Target Behavior Coding Sheet (see appendix 1)

The artsJAM director and social worker identified target behaviors prior to the start of the program. The target behaviors list was also influenced by Junior Achievement's list of Life Skills, which include behaviors that Junior Achievement consider important to job performance and social integration. The target behaviors are formatted into a functional observation sheet with categories for the target behavior, positive and negative expressions of the behavior, and researcher comments. The positive and negative behavior category is divided into quarters, which represent 15-minute intervals.  The target behaviors include:

·          Listens

·          Follow Directions

·          Focuses Attention

·          Body Language

·          Voice Tone

·          Accepts Criticism

·          Accepts Differing Opinions

·          Gives Constructive Criticism

·          Shares Supplies

·          Cleans Own Mess

·          Actively Participates

Student Goal Sheet (see appendix 2)

The Student Goal Sheet is intended to start a conversation with the student about the goals of the artsJAM program, appropriate work behaviors and the student's personal strengths and weaknesses. At the beginning of the program each student is asked to identify three things to work on during the program, as well as strategies the student will use to achieve each of their goals.

Student Goal Sheet Final Review (see appendix 5)

At the end of the program, the artsJAM director and social worker interview students again. This interview is intended to assess whether or not the student achieved his/her goals and to debrief the artsJAM experience. 

Student Weekly Assessment (see appendix 4)

Students complete the weekly assessment at the end of each work week. This assessment is intended as a simple way for students to reflect on their behavior and track their progress toward meeting their goals.  The assessment specifically asks students to identify weekly successes and to identify challenges that still need to be addressed.

Student Daily Assessment (see appendix 3)

The management team created a daily assessment that can be quickly completed by the students. The assessment includes three categories:

·          Getting along with others

·          Active participation

·          Challenge myself to do my best

Students use colored stickers to identify if their behavior is good (green), needs improvement (yellow), or is unacceptable (red).  Because the assessment sheet includes a row for each workday, the student and social worker can visually identify trends in behavior throughout the program.

Parent Survey

The management team expressed concern about parents completing and returning surveys. The management team, therefore, decided that it would be more effective to survey parents at the culminating event. The researcher prepared two open-ended questions and surveyed parents as they entered the culminating event and during the event afterglow. The two questions were:

1. Have you noticed any effects of the artsJAM program on your child? Describe.

2. Do you have any concerns about the program or suggestions for improvement?

Work Product

The students created books during the morning work session. The books are creative products that involved making artist paper for the book cover, cutting pages, formatting, binding the books, and packaging them for retail sale. This work was accomplished under the direction of a visual artist in a simulated work environment. The students created over 600 books of varying shapes, sizes and designs.

Procedures

Target Behaviors

Students were video taped for one hour during the morning work session for two consecutive days at the beginning and at the end of the program. Coding Sheets were used to score each student's expression of the target behaviors. Each target behavior was recorded as pre or post variables in an SPSS database (statistical analysis software). The mean of each pre variable was matched with its corresponding post variable and a Paired Sample T-Test was used to analyze increases or decreases in the target behavior at the end of the program. In this study, significance is identified at less than .06.

Interrater reliability. Two raters independently watched the video taped sessions and scored each of the pre and post target behavior variables. Variables were scored as 1=positive expression of the behavior, 0 as acceptable behavior, and -1 as a negative expression of the target behavior. Agreement was 97.1% or higher for each variable and 98.0%for total combined pre and post behaviors. Any discrepancies were resolved through discussion with the lead researcher.

Student Goal Sheet, Weekly Assessment and Final Review

Thematic exploration was used to consolidate, chart and compile emerging themes identified on the Student Goal Sheets, Weekly Assessments and the Goal Sheet Final Reviews. 

Student Daily Assessment

The Student Daily Assessment data was used to create a time study to investigate changes in student self-reported behaviors. The results of the time study were then compared to the results of the Target Behavior analysis to identify correlations or discrepancies.

Parent Survey

Thematic exploration was used to consolidate, chart and compile emerging themes identified in the Parent Survey data.

Work Product

One of the artsJAM program goals was to help prepare students for the world of work. Since the video taped observations took place during the morning simulated work sessions, it seems appropriate to examine student productivity during these sessions. The artsJAM director identified a goal of 100-200 packaged products by the completion of the program to the lead researcher before program began.

II. FINDINGS

Target Behaviors

A paired T-test analysis revealed that there was no significant increase or decrease in the mean of six of the target variables. These variables include:

·          Listens

·          Follows Directions

·          Voice Tone

·          Accepts Criticism

·          Cleans own Mess

·          Actively Participates

Although there was no significant change in these target behaviors, it is interesting to note that there was a high level of acceptable behavior displayed by the students in all of the target behaviors from the beginning of the program. Acceptable behavior percentages for each variable ranged from 95.24% to 97.81%.

Table 7

*There were no positive or negative expressions of behavior for this variable. No situations arose during the video taped sessions that required the students to accept, or not accept, differing opinions.

Four of the paired variables did show a significant change in behavior. Three of the paired variables (focuses attention, body language, and gives constructive criticism) show a small, but negative increase in the target behavior.

Focuses Attention

The focuses attention variable is at the borderline of the study's established significance level of .06. The overwhelming majority of students displayed acceptable behavior in the pre and post examination and students stayed focused just over 2% less at the end of the program than when the program began.

 Table 8

 Table 9

Body Language

Students used acceptable body language 97.35% of the time. Body language was, however, 2.57% worse at the end of the program in comparison with student body language at the beginning of the program.

  Table 10

 Table 11

Gives Constructive Criticism

Because there was not much group discussion during the observation periods, student criticism was only observed in casual discuss while students worked. Student discussions remain appropriate 97.35% of the time. Negative criticism, although infrequent, increased by 2.66%.

Table 12

 Table 13

Shares Supplies

The shares supplies variable is the only variable that met statistical significance and showed a positive increase. At 3.4%, this variable also had the highest level of change, positive or negative. Again, a high percentage of students displayed appropriate behavior throughout the program.

Table 14

 Table 15

All Variables from two-day Observation Pre & All Variables from two-day Observation Post

Although the data clearly shows that there was not a significant change in student overall behavior, it is important to note a few of the statistics. First, there was a small negative increase in behaviors. Because the significance level is at .228, no claim of change can be made. The fact that students displayed acceptable behavior 97.35% of the time, on the other hand, is without dispute.

Table 16

Table 17

Student Goals

The project director and social worker indicated that most of the students seemed nervous about the goal setting process. Goal setting was facilitated through a private interview with each student. It is possible that the students felt apprehensive about the process because it was called an "interview" and they were concerned that if they did not pass their interview, they would be removed form the program. It is also possible that the formal nature of the process intimidated them. The students were aware that two adults would be questioning them during the interview and that the interviews would be conducted in a private office with the door closed.

There are possible positive and negative relationships that one could support between student nervousness and interview performance. For example, because the process was formal, students seemed to perceive the process as something very serious. This perception could lead to greater student reflection and preparedness. On the other hand, the project director and social worker reported that some students were reluctant to make any comments and in some cases they had to help students formulate their responses. This could lead to a lack of understanding and ownership of the student's goals. A lack of understanding and ownership of their goals could make the student less likely to achieve these goals.

All of the students had seemingly appropriate goals identified and strategies to achieve the goals during the program. Most of the identified goals are behavioral and fall into three categories: appropriate participation and communication. A majority of students also consistently reflect these goals in the statements they make on the weekly self-assessments. 20% students identified goals that were in no way reflected through their weekly assessment comments.

Student Daily Self-Assessment

The time study of the self reported student daily self-assessment revealed similar findings to those observed in the analysis of the target behaviors. A large percentage of students reported their behavior as "good" in the three categories: getting along with others, active participation, and challenge myself to do my best. Furthermore, there was no significant change in student self-reporting over the course of the program.

Seven cases in which students rated their behavior in the three categories as "good." Three of these cases match observed behavior from the coded target behaviors. Four of the cases are not congruent with the target behavior observations.

The Student Daily Assessment data was used to create a time study to investigate changes in student self-reported behaviors. The results of the time study were then compared to the results of the Target Behavior analysis to identify correlations or discrepancies.

Student Weekly Self-Assessment

80% of students provided appropriate weekly comments. Appropriate comments are defined as statements that either reflects the student's initial goals or address specific challenges the student faced during the week. The most common statements identified specific behavioral issues that arose during the week. Below is a typical example of this.

·          These are the things that went well for me today:

I was not talking about other people.

·          Things I still need to work on:

I talk alot.

20% of students wrote statements that did not reflect their goals or challenges.  For example, to the question "These are the things that went well for me this week," these students responded with comments such as

·          My best friend is ______ & ______

·          We will go to arts

·          I had a good time coming back home

·          Everything went well (repeated every week)

Parent Comments

Family members were surveyed as they entered the culminating event or during the afterglow of the culminating event. Every student had at least one parent or family member in attendance at this event.

100% of family members indicated that the program had a positive effect on the participating student. 21% of those were so enthusiastic in their support for the program that they were almost brought to tears.  All of the parent comments divided into two main categories: affective or practical. Affective comments included a variety of descriptions indicating a student's greater sense of self. Parents expressed this idea of self-concept in a variety of ways. The comments included such ideas as

·          Child has greater self-awareness

·          Child is proud of him/herself

·          Child is more confident

·          Child is more independent

·          Child is more responsible

To more clearly illustrate this point it may be helpful to view a few parent quotes. The following is a small sample of specific feedback received in regard to student self-concept.

"I have witnessed my child become more aware of his surroundings."

"I have seen (my child) take an interest in his personal appearance."

"To say that this program is a gem is not at all true it is a Godsend. VSA has given my child an increased level of self-esteem and a sense of peace."

"Through VSA, I saw (my child) evolve into a more articulate, independent young teenager, whose social interaction with his peers, developed more than at any other point in his young life."

"I am so proud of my son I really did not think that he would have got as far as he did."

The last quote indicates an unexpected finding that was expressed by 33% of those surveyed. These family members indicated that they did realize that the participant was capable of some of the things that he/she had achieved. One parent, for example, was initially concerned about their child navigating the public transportation system to travel to and from the program site. Another parent did not expect their child to be able to work with others in a productive way throughout the program. 40% of respondents described their surprise at the high level quality in the student's products.

There was one recurring practical theme that emerged from the parent comments.

67% of the parents described extended benefits to their child being paid for their work in the program. Three parents indicated that it provided them an opportunity to work with their child on money management skills.

Only one parent offered a suggestion for improvement of the program. The parent hoped that some transportation assistance could be provided. None of the respondents expressed concerns or criticisms of the program.

III. Conclusions

As was stated in the introduction, the primary goal of this study is to provide VSA arts of Michigan an evaluation of the artsJAM program's ability to develop the skills necessary to help students with disabilities transition to the world of work.

The researcher and the management team did not anticipate the high percentage of students that would meet the target behaviors at the beginning of the program.  Although the study did not find significant positive increases in the student target behaviors, it is notable that the students were able to maintain appropriate behavior on all of the target behaviors at least 95% of the time. This phenomenon may be attributed to several factors, including the application process, the fact that the program paid students for their work, goal setting, or the expectations set out at the beginning of the program. More study is needed to delineate a cause and effect relationship between the program and any one of these factors.

Students also displayed a small increase in negative behavior on three of the target behaviors: focuses attention, body language, and gives constructive criticism. This negative change may be related to an observation made by both of the independent raters. Both raters commented to the lead researcher that there was a noticeable difference in classroom management during the taped sessions near the end of the program. For example, several students sat for nearly 20 minutes without the materials needed to begin work at the beginning of taped session 3.  Also, at the end of the day 3 and 4 work sessions, several students completed their work, cleaned up, and were not sure what they should be doing. The program director also commented to the lead researcher that there was less organization during the morning work sessions toward the end of the program.

Despite the three small increases in negative behavior, when the combined variables were analyzed, the overall change in the target behaviors was not statistically significant. The researcher did see, however, some change in student behavior.

Student self-awareness emerged as an important theme in the student goal sheets, weekly self-assessments, the daily self-assessments and in the parent comments. Prochaska (Prochaska, DiClemente, & Norcross, 1992; Prochaska, Norcross, & DiClemente, 1994; Prochaska, Redding, & Evers, 1997) studied the connection between self-awareness and behavior change for over 2 decades and developed the Transtheoretical Model. Their work revealed that behavior change evolves through different stages:

Stage 1: Precontemplation

Individuals in this stage do not believe they have a problem and have often constructed defenses that aid in denial of the problem.

Stage 2: Contemplation

Individuals acknowledge having a problem and begin to deliberately increase awareness and knowledge related to the problem.

Stage 3: Preparation

Before initiating behavior change, individuals should reevaluate themselves with respect to the problem, develop commitment to change, and construct a detailed plan for change. By the time they reach this stage, individuals begin to perceive greater benefits than barriers to change.

Stage 4: Action

Behavior change is initiated. Others are likely to recognize a person's progress toward change. After at least 6 months in the action stage, the person may move into the fifth stage.

Stage 5: Maintenance

Though change is maintained more easily now, some vigilance is still required to avoid slips or setbacks. If and when the change becomes so automatic that there is no possibility of reverting to a former behavior, the goal--"Termination"--is reached.

Although somewhat anecdotal, the data collected seems to indicate that students are at various stages of the Transtheoretical model.   For example, 20% of participants increased their use of the "needs improvement" and "unacceptable" codes on their daily behavior in the second half of the program, although the taped observations did not support this change. This seems to indicate that the student's perception of his or her own behavior changed over the course of the program. If this is true, it follows that the student experienced an increase in awareness of his or her behavior. 60% of participants' daily self-assessment was congruent with their behavior from the video taped sessions. This seems to indicate that the majority of students have an accurate perception of their behavior.

Another indicator of an increase in self-awareness can be found in many of the parent comments. The two comments listed below seem to clearly indicate this awareness.

"I have witnessed my child become more aware of his surroundings."

"I have seen (my child) take an interest in his personal appearance."

The second comment also implies a behavioral change on the part of the student. The parent goes on to explain that the child bathes more often and has become highly conscious of his or her clothing. According to Prochaska, this student is already in stage four of the Transtheoretical model: Action.

20% of participants did not demonstrate the same level of self-reflection on the weekly self-assessment as the rest of the students in the program. For example, to the question "These are the things that went well for me this week," these students responded with comments such as

·          My best friend is ______ & ______

·          We will go to arts

·          I had a good time coming back home

·          Everything went well (repeated every week)

It would be helpful to know if this lack of self-awareness is related to the participant's disability. Unfortunately, VSA was not provided with this information. Therefore, it is not appropriate to speculate about possible relationships.

Work Product

There is one strong indicator of that the artsJAM program met its intended goal. It is the fact that students were highly productive during the morning work sessions. Before the program began, the program director indicated to the lead researcher that she hoped the students would create 100-200 books during the work sessions. By the end of the program, however, the students had produced over 600 well-crafted books and had them packaged for retail sale. This is a 200% productivity increase over the original target goal.

IV. Recommendations

Many aspects of the artsJAM Detroit! Work Alternatives for Youth program are successful. The application and interview process, the student goal sheets, the student weekly self-assessment and the student daily self-assessment seem to generate high expectations and create an atmosphere of accountability that keep students focused. The tools also require students to do a considerable amount of self-reflection that seems to have contributed to increasing or maintaining appropriate behavior throughout the program.

Considering this focus on self-reflection and what is known about the precursors to behavior change, it is recommended that the target behaviors be re-assessed. In addition, students did not often have the opportunity to demonstrate some of the target behaviors. It would, therefore, be advisable to cultivate a new list of target behaviors that identify student self-reflection and appropriate participation in their work environment.

One question that immerged during this study is why some students did not seem to demonstrate self-reflection. It would be helpful to identify each participant's disability before the beginning of the program. This research could yield important information about relationships between student successes and disabilities. This information could also help the program director and instructors make accommodations that might lead students with particular disabilities to become more successful.

There were many aspects of the artsJAM program that were not studied. The Junior Achievement activities and "Circle Time" are two elements that may have had significant impact on student self-awareness, behavior change and productivity. Expanding the study to include the impact of these components, as well as the impact of the various arts activities, would help develop a clearer picture of the strengths and weaknesses of the program. Expanding the study would also provide the researcher the opportunity to factor out influences of other parts of the program on the results defined by this study. This would help substantiate the effectiveness of the tools used in this study.

It became clear through an analysis of the data that the participants are working at dramatically different levels.  Some students seem to possess the personal qualities and skills to transition to the workplace, while others may still need considerable training and experience. If the ultimate goal of the program is to help students transition to the world of work, it may be advisable to create two programs. The first program could be similar to the current artsJAM program. It would offer students a broad range of experiences and focus on self-reflection and developing appropriate behavior. The second program could be for students that have proven themselves reflective and responsible. These students could work on specific higher-level skills that they will need in the workplace. VSA could also facilitate internships or work placement as a culmination of this program.