Data Analysis from The Document "The Evaluation System of English Teaching in the Implementation of School Based Curriculum (SBC) for Vocational School"

 

4.1.2        Data Analysis from the Document

The document analysis was conducted to find out how the teachers do the evaluation from the analysis of the relevant documents.  The documents which were analyzed in this research were the SBC document, English Syllabus, Lesson Plan, MMC (the Mastery Criteria), the score lists, and a sample of Final test.

 

4.1.2.1  SBC document of SMK Merdeka

SBC SMK Merdeka has two documents, namely document I entitled ‘SBC SMK Merdeka’. Document II consists of the syllabus and lesson plan of each subjects  taught in SMK Merdeka.

Here, the document which is elaborated is dealing with document I. For document II, it is presented in the syllabus and RPP  in later discussion. Each department in the site has its own SBC. Therefore, there are two SBC documents respectively SBC for automotive and mechanics departments. Basically, the two documents have similar structure in terms of the content. The difference is in the vocational skill competence as the major subjects.

In general, the SBC document  of the school is divided into some chapters:

Chapter I the Objective of Vocational Education

Chapter II The Vision and the Mission

Chapter III the Objective of SMK Merdeka Soreang

Chapter IV the Objective of Vocational Program ( such as Automotive Department)

Chapter V Competency Standard

A.      Graduate Competence Standard of Vocational School

B.      Graduate Competence Standard of each subjects such as  PAI, PKn, Bahasa Indonesia, Penjasorkes, Seni Budaya, Bahasa Inggris, Matematika, IPA, Fisika,Kimia, IPS, KKPI, Kewirausahaan, and Vocational Subjects.

C.      Competency Standard and Basic Competence Standard of all subjects taught in the school.

D.      Competency Standard and Basic Competence Standard for Local content (Mulok –Bahasa Sunda)

Chapter VI  Curriculum Structure

Chapter VII  Academic Calendar.

Compared to the guideline from BSNP (2006), the SBC has not had the Introduction chapter which elaborates about the rationale, foundation and the objective of designing SBC, as  the government (BSNP: 2006) states that the structure of SBC should consist of :

Chapter  I. Introduction (consisting Rationale, Foundation and Objective)

Chapter  II. The Aim /Objective of SBC (Consisting Vision, Mission and Objective)

Chapter  III The Structure and  Composition of SBC

Chapter  IV  Academic Calendar

Chapter   V  Conclusion

The Appendices (such as Yearly Program, Semester Program, syllabus, teaching  plans, indicators, etc).

 

In SBC document at the site specifically, it is stated the objective of education in the vocational school. The objective is similar to what BSNP  (2006)  has stated that the objective of  education in vocational school is to enhance intelligence, knowledge, personality, good morale, and skills to  live independently and or to continue further education according to their vocational skills.

The objectives of the vocational education in the research site  can be described as follows:

a.       Preparing the students to be a productive and  autonomous person to fill the  job opportunity  as the intermediate mechanics  according to his skill.

b.      Supplying the students with skills in engineering and service to be a professional, productive and autonomous mechanics in meeting the demand of  work field.

c.       Developing  the entrepreneurship ability in facing  the globalization era and free trade era.

d.      Supplying the students with science and technology to continue  their further  study.

e.       Supplying the skills in communicating using English.

 

It is stated clearly that English has been a priority as it is stated in the objectives. It is in accordance with the SKL (Government Rule No.23/2006) that the graduates are hoped to have capability in communicating with English.

There are  graduate competencies standard (SKL), Competencies Standard (CS) and Basic Competencies (BC) as the points to be reached for the students in vocational school which the content are exactly the same as what the government has been regulated. In term of CS of English, teaching of English is mainly aimed at making the students in SMK  have an ability in communicating with English in  novice level,  elementary  and   intermediate level. Those competencies are then defined into some basic competencies (BC). CS/BC gives the direction of the teaching and learning process to be. It is the teacher role to develop these  basic standards to be implemented in the teaching learning process and  also to develop some assessments to measure the students’ achievement to reach the competencies. One of the indicators of successful teaching learning is through looking at the indicators  of each BC whether  it is fully mastered or  not.  In term the curriculum structure, English is taught for 440 meeting hours out of the total 4602 hours    in a year in which for each meeting is given 4 hours. It is in line with what BSNP (2006) has stated.

As it is mentioned before, there are 3 Competencies Standard ( CS- for short)  required for the students in the vocational school to achieve , namely:

1.      To communicate with English in the  novice level ( for the first grade).

2.      To communicate with English in the  elementary level ( for the second grade).

3.    To communicate  with English in the  elementary level ( for the third grade)

The competencies standard are then more clearly defined  in the BC. Each  grade has certain BCs, as for example, for the first grade, the  BC such as understanding  some basic expression in social interaction in the daily life which deals with greeting and leave taking, introducing,  apologizing, how to tell the time, person characteristics etc; explaining the activities in progress, making simple invitation, and many others.

In the second grade, the students are required to have typical basic language competencies include understanding simple conversation, taking a message, telling the past  action and  the plan in the future, etc.  For the third  grade , the competencies deal with  understanding monolog in the work place, making a report, understanding manuals, making  business letters,  presenting the report, etc.

Detail explanation is described  in  table below :

 

Table 4.1    CB/BC  for Vocational School (BSNP:2006)

 

Level	CS	BC
First Grade	Berkomunikasi dengan Bahasa Inggris setara Level Novice	1. 1   Memahami ungkapan-ungkapan dasar pada interaksi sosial untuk kepentingan kehidupan 1. 2   Menyebutkan benda-benda, orang, ciri-ciri, waktu, hari, bulan, dan tahun 1. 3   Mendeskripsikan benda-benda, orang, ciri-ciri, waktu, hari, bulan, dan tahun 1. 4   Menghasilkan tuturan sederhana yang cukup untuk fungsi-fungsi dasar 1. 5   Menjelaskan secara sederhana kegiatan yang sedang terjadi 1. 6   Memahami memo dan menu sederhana, jadwal perjalanan kendaraan umum, dan rambu-rambu lalu lintas 1. 7   Memahami kata-kata dan istilah asing serta kalimat sederhana berdasarkan rumus 1. 8   Menuliskan undangan sederhana
Second Grade	Berkomunikasi dengan Bahasa Inggris setara Level  elementary	2.1.   Memahami percakapan sederhana sehari-hari baik dalam konteks profesional maupun pribadi dengan orang bukan penutur asli 2.2.   Mencatat pesan-pesan sederhana baik dalam interaksi langsung maupun melalui alat 2.3.   Merinci tugas pekerjaan dan latar belakang pendidikan yang dimilikinya secara lisan dan tulisan 2.4.   Menceritakan pekerjaan di masa lalu dan rencana kerja yang akan datang 2.5.   Mengungkapkan berbagai macam maksud hati 2.6.   Memahami instruksi-instruksi sederhana 2.7.   Membuat pesan-pesan pendek, petunjuk dan daftar dengan pilihan kata, ejaan dan tata tulis yang berterima
Third  grade	Berkomunikasi dengan Bahasa Inggris setara Level Intermediate	3.1     Memahami monolog yang muncul pada situasi kerja tertentu 3.2     Memahami percakapan terbatas dengan penutur asli 3.3     Menyajikan laporan 3.4     Memahami manual penggunaan peralatan 3.5     Memahami surat-surat bisnis sederhana 3.6     Memahami dokumen-dokumen teknis 3.7     Menulis surat bisnis dan laporan sederhana

 

Looking at the content of CB and BC, it can be said that the competencies that the students  should be acquired relate to the use of English in everyday life. The texts are more focused on the simple functional texts such as manuals, simple report, short messages, business letters, etc. English in the vocational school is much   simpler  than English in the public school. The aim of teaching in the vocational school tends to make  students develop their  knowledge, intelligence, character,  good behavior,  and skills to  live independently and or to continue  the further study according to their vocational programs (SBC SMK Merdeka, Tujuan SMK, 2010).

Taken as an example,  the CB/BC for the third grade. As it is outlined by the SBC Curriculum, for the third grade students, the level competencies is  intermediate. It is expected that the students in this grade have higher knowledge, skills and attitude in using English than those in the first and the second grade. As it is mentioned earlier, in this grade, the required competencies including understanding monolog in the work place, making a report, understanding manuals, making business letters, presenting the report, understanding technical documents. Those competencies can be found in the TOEIC test. As it is indicated in the headmaster’s interview (see Appendix 4), especially for the third grade students, they are required to follow TOEIC at the end of the study. Therefore, the school has carried out some suitable efforts to achieve the basic competencies such as by administering the TOEIC Test for the students.

There are  seven BCs to achieve in the third grade. In the BC 3.2, for example,   the students are required to understand simple conversation with the native speaker of English. It is clearly defined in the indicators that the students are expected to understand some expression used by the native speakers in making reservation in a hotel, restaurant or travel agent; expressing complaints, making arrangements, confirmations, cancellations, imaginations and interviews. They are hoped to be able to understand the relevant expressions used in those situation and those purposes. The assessment deals with oral and written assessment include  role play the dialogues, completing the dialogues and multiple choice items.

In short, based on the analysis on the SBC document in that school, it can be viewed that the document is in line with the points of BSNP (2006) on developing School Based Curriculum and Government Rule no.23/2006 about the competencies to be reached for the students at vocational education.

 

4.1.2.2  The English Syllabus

 

The English syllabus in Indonesia is competence based syllabus. The content of syllabus is the same as it is outlined by the government (BSNP, 2006). It consists of the name of the school, the name of the subject, class and semester, CS and BC, and allocation in the Title heading. On the table, it is written BC, indicators, materials, learning process, assessment, allocation, and material source. The complete portrait of the syllabus can be found  in the Appendix 7. In the teaching implementation, the syllabus is described in the lesson plan which is then implemented in the teaching learning process. It is evaluated and the teacher can change or develop the syllabus based on the need. In addition, the syllabus is in line with Richard and Rodgers (2001) description that in competency based teaching, a syllabus and the course content are then developed around the subject. Competencies consist of a description of the essential skills, knowledge, attitudes, and behaviors required for  effective performance  of a real world task or activity.

 

4.1.2.3  Lesson Plan (RPP)

 

The teachers usually  make their lesson plan before teaching or  in the beginning of the academic year. The school officer have often assigned all the teachers to create their own administration. The format of the administration is often discussed in the IHT (in house training) which is usually conducted at the beginning of the new academic year. Therefore, in term of the administration of  syllabus, RPP, and MMC, it can be viewed that the English teachers have designed them well as similar to BSNP (2006). Also, it is one of the a good characteristics that the teacher should design the lesson plans effectively (Brown, 2001:430). The example of the teacher RPP can be seen in the Appendix 9. As it is stated in the sample of a teacher’s lesson plan, it can be found that the allocation is not suitable with the time allocated (22 teaching hours). The lesson activity is appropriate for 2-4  teaching hours. Therefore, the  lesson plan needs to be revised.

 

4.1.2.4  MMC  (Kriteria Ketuntasan Minimal- the Minimal Mastery Criteria)

The  teachers  have made the MMC at the beginning of the teaching year. For the 2010/2011 especially for English, the MMC value can be presented as follows:

 

Table 4.2  The MMC Value  for all grades

 

Grade	MMC
First  Grade	60
Second  Grade	60
Third Grade	61

 

The final result of the MMC score is derived from the calculation of the three aspects, intake, facility, complexity. The detail process of the calculation can be seen in the appendix 10. In the upcoming new academic year, the MMC value is increasing to be  65 for all classes. The process in designing MMC or mastery criteria is in line with  BSNP (2006) in the previous section. The flow of activities in designing MMC is the same as it is outlined by BSNP that is determining  MMC indicators, MMC Basic Competences, MMC  Competence Standard, and MMC Subject Matter.

 

4.1.3. 5 The Students’ Score Lists

The score lists are taken from the three respondents, as described below.

4.1.3.5.1 The  Score  list from the Respondent 1

The teacher taught at the three classes, X TKR 2, X TKR 5, and XII TKR 3. In her evaluation system, the teacher takes  into account the students’ daily tests, the assignments, the mid and the final test scores as well as the attitude/ behavior , and the presence.

In her list, it is described  scores that the students have gained and the attendance list throughout the whole year. The distribution of the scores then was compared with the MMC value as seen in  Table 4.3 below. It can be noted that the scores in the second semester had not fully filled. Some students had blank scores. But for the final score in the rapport book considered many aspects including all the gained scores in the daily assignments, the tests, attitude and presence. When a student  does not fulfill  all aspects of the evaluation,  there will be a discussion in a meeting board for further decision whether he  can pass  the grade or fail. The meeting consists of the teachers, the homeroom teacher  and the school officer.  It is in line with Grolund’s idea that evaluation is a whole range of activities to find, analyze, and interpret the process and  learning outcome data which is  conducted systematically and continually in order to get meaningful information in making decision (Gronlund, 1985). Similarly, Rudiana (2006: 123) points out that evaluation is very essential for teachers to know their students exactly and to enhance learning outcome. Additionally, Herrick (1996) maintains that a unitary system in which a key to a unitary system is an assessment system that measures not only knowledge and facts, but also the application of knowledge as well as the impact of experiences from work and life. This assessment is in accordance with the nature of evaluation based on SBC in which the assessment should include cognitive dealing with the students’ knowledge, affective  regarding to the attitude or behavior and psychomotor relating to the skill.

 

4.1.3.5.2        The  Score  list from the Respondent 2

The teacher taught English at  XII TKR 2, XI TKR 1, X TKR 3. In his list, it is described the scores that the students had  gained from the daily tasks, assignments and the tests. In his presence, it is stated the date of the assignments. The assignments were marked by the date on the score space available. There are a lot of dates in the presence list representing the fulfillment of the daily tasks. The   description of the scores on the daily, mid and final tests can be seen in the Table 4.3.

As mentioned earlier, according to this respondent, the evaluation is a process based. Therefore, the final score is determined by the students’ motivation in doing the assignments or the daily activity in the class which had been assigned  by the teacher. Whenever the students respond his tasks well, the score is good as well, and vice versa. The evaluation is in accordance with what Gronlund (1985), Rudiana (2006), SBC (BSNP:2006) and Herrick (1996) have maintained that  the assessment should include a whole range activities to find, analyze, and interpret all aspects of the students get meaningful information in making decision.

 

4.1.3.5.3        The  Score  list from the Respondent 3

The teacher taught at one class only, that is X TM 1. The score list describes all the students’ scores on the daily assignments, and the tests. It can be noted that generally the scores in all aspects are above the MMC. Therefore, the evaluation of the class according to the respondent is very satisfying. The respondent’ evaluation can be viewed as in accordance with what Gronlund (1985), Rudiana (2006), SBC (BSNP:2006) and Herrick (1996)  as discussed earlier.

In making a clear description, Table 4.3 below describes the students’ scores mean compared with the MMC value:

                     Table 4.3 The students’ scores mean compared with the MMC value:

 

The Source of Data	The Class	Semester	MMC value	Daily test	Assignment	Mid term test	Final test	Presence	Additional information
R1	X TKR 2	1	60	Above	Above	Below	Below	Mostly present	Final test score is less than mid term
		2	60	Above	Above	Below	Below	Mostly present	Final test score is similar to mid term
	X TKR 5	1	60	Above	Above	Below	Below	Mostly present	Final test score is less than mid term
		2	60	Above	Above	Below	Below	Mostly present	Final test score is similar to mid term
	XII TKR 3	1	61	-	Above	Above	Below	Mostly present	Final test score is less than mid term
		2	61	-	-	-	-	Mostly present	Enrichment or accelerated learning
R 2	X TKR 3	1	60	Below	Above	Below	Below	Mostly present	Final test score is similar to mid term
		2	60	Above	Above	Above	Above	Mostly present	The final and the mid term are better than the last semester
	XI TKR 1	1	60	Above	Above	Above	Above	Mostly present	The scores are good and mostly above the MMC
		2	60	Below	Above	Below	Above	Mostly present	Final test score is better than mid term
	XII TKR 2	1	61	Above	Above	Below	Above	Mostly present	Final test score is better than mid term
		2	61	-	-	-	Below	Mostly present	TO test as the Final test
R 3	X TM 1	1	60	Above	Above	Above	Above	Mostly present	The scores are above the MMC in general
		2	60	Above	Above	Above	Above	Mostly present

 

From  Table 4.3 above, it can be concluded that the scores on the midterm and the final test are mostly less than the daily tests or assignments. Taken as an example, for class X, the final test score is usually better than the midterm test. Meanwhile for class XI, on the second semester, the final test is better than the midterm test. For  class XII, the scores record is not completed  because the students began to have an enrichment teaching as the preparation for the upcoming National Exam. The students’ presence was written in a certain attendance list which was collected to the school officers. In general, the students were present in the overall process of teaching learning in the class. Some students who got problems in the final score because of the inadequate scores or the presence, were discussed in a meeting involved all the school personnel to decide the final decision whether they can pass the grade or be fail.

Thus, in general, the teachers evaluation include internal and external evaluation as suggested by Heaton (1995). Types of   the evaluation are in line with what can be categorized by Grondlund (1985) namely placement evaluation, formative evaluation, diagnostic evaluation, summative evaluation.

 

4.1.3.5.5        Analysis  on the Test Item Sample

 

The test item analysis is taken from the analysis of  the latest English UKK ( Ujian Kenaikan Kelas/ Grade promotion exam) for the first grade ( 2010/2011).  The test was conducted in June 8, 2011 which contains 40 multiple choice items. The listening test is not included into test, all items are written. The test item consists some indicators that are presented in the elementary level especially for the second semester. Therefore the content is suitable, or valid as it measures what it is intended to measure (Heaton,1995; Hughes , 2003; Gronlund,1985; Sudjana,2006; Fulcher,2007; BSNP,2006).

In detail the test mapping can be shown in  Table 4.4 below :

Table 4.4 The Test Mapping

 

CS/BS	Material	No Item on the Test	Total Number
1.5   Menjelaskan secara sederhana kegiatan yang sedang terjadi	·   Words and expressions used in the context of telling or describing events. ·    Grammar: Present continuous, future continuous. ·    Sentences using ‘there +be’ ·    Prepositions: in, on, at, under, etc. ·    Questions about events: ·    Expessions of feelings / opinions concerning an event.	1,2   3,4,5 6 7,8, 9 10,	10
1.6   Memahami memo dan menu sederhana, jadwal perjalanan kendaraan umum, dan rambu-rambu lalu lintas	·   Samples of memo ·   Sample of menu ·   Words and expressions to explain signs and symbols: ·       Samples of time table and schedule ·       Degrees of comparison: ·       Pronouns and Reported Speech	11,12,13   14,15 16,17,18   19,20,21,22	12
1.7   Memahami kata-kata dan istilah asing serta kalimat sederhana berdasarkan rumus	·   Words and expressions used in expressing preference: ·    Conditional sentence type1: ·    Words and expressing used in aCSing for and giving direction (location): ·    Prepositions of place: in front of, behind, beside,etc	24,25 23,28   26,27 29,30	8
1.8   Menulis  undangan sederhana	1.8.1       Grammar review: “will”, “could”, “would” and prepositions: in, on at. 1.8.2       Samples of invitation (personal invitation). 1.8.3       Parts of personal invitations. 1.8.4       Contents, style, spelling and punctuation.	31,32,33   40 34-39	10
The Total Item			40

 

            The test was administered on the second semester so the materials on the test cover  all the BC in the semester. The test form can be found in the Appendix 11. The test  was carried out by all the students in each grade at SMK Merdeka Soreang. The test consisted of two types, Packet A and Packet  B. The items in each packet are the same, but the difference is in the order of the numbers for each items. To specify the analysis, the Packet A is chosen. The test papers are  from the two classes of the first grade students.  The total number of the students is 40  students. The item analysis can be seen in the Appendix 7.

The item analysis followed  these steps (Heaton :1988):

1.      Arranging the obtained data  starting from the  lowest  to the highest score

2.      Separating the Highest score into the Upper Class (U) and the Lowest score into the Lower Class (L)  through  taking out 27 % of  the papers.

Upper   Class (U) = 27 % X 40 =  10.8 →11

Lower  Class  (L) = 27 % X 37 =  10.8 →11

       For the right answer, it is scored 1. For the wrong answer it is scored 0. After all the students’ answers  have been counted, then  the Item difficulty and the discriminating       power can be calculated.

3.      Determining  item difficulty (ID)  using the formula (Heaton :1988):

ID = (U+L)

            T

 ID – item difficulty level

  U  - right answer  from the Upper class

  L  - right answer from the Lower Class

  T  - the total number of the students  from Upper and Lower Classes.

Criteria  for  difficulty level :

ID	≤  0,24     = the item test is difficult
0,24   ≤  ID≤   0,75     = the item test is moderate
ID	≥ 0,75     = the item test is easy

4.      Determining  the   Discriminating Power  using the formula (Heaton :1988):

 

DP = U-L

           ½  T

 

DP = Discriminating Power

  U  - right answer  from the Upper class

  L  - right answer from the Lower Class

  T  - the total number of the students  from Upper and Lower Classes

 

Criteria  for Discriminating Power (DP) :

DP    ≥ 0,40    = the item can  be used
0,20 ≤  DP ≤ 0,39     =  the item should be revised
DP    ≤  0,19      =  the item should be changed/discarded

 

From Tables 4.4  in the Appendix 7, in term of the difficultness of the item, it is shown that the items have some criteria:

Table 4.5  The Result of  the Item Difficultness

 

Criteria	Item Number	Total
Easy	1,5,7,9,11,14,17,21,23,29,32,34,40	13
Moderate	2,3,4,6,12,13,15,16,18,19,20,22,24,25,26,27,28,31,33,37,38,39	22
Difficult	8,10,30,35,36	5
Total Item		40

 

It can be said that for most of the students, the test is considered moderate (Suharsimi,1993) as it is indicated by  the number of  moderate items based on the calculation (22 items), whereas 13 items are easy and 5 items are difficult.

In term of Discriminating power, it can be shown on the Table below:

Table 4.6  The Discriminating Power Analysis

 

Criteria of  the Item	Item Number	Total
Can be used	1,2,3,4,5,6,7,9,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,36,37,38,39,40	37
Should be revised	8	1
Should be changed	10,35	2
Total Item		40

 

It indicates that  the test is well-structured (Heaton :1988) as it can determine between the upper class students with the lower class in a large number of the item used. Only 1 item needs to be revised in terms of the test construction and the structure of the option There  are also 2 items  considered to be ‘bad’ item and they need to be discarded or changed.

 

5.      Estimating the Reliability using the formula (Hatch and Lazaraton, 1991; Hatch and Farhady,1982) :

R = 1- M (n-M)    (Kuder –Richardson Formula )

               ns²

R = reliability

M = mean

n = number of the students

s = standard deviation

 

The process of the calculation can be presented in the Appendix 8. As the calculation showed that the reliability index is 1.05, so the test is considered to have a high reliability (Hatch and Lazaraton, 1991; Hatch and Farhady,1982). It is stated that the test is said to be reliable when the tester has constructed, administered, and scored the  test in a such way that the scores  actually obtained on a test on a particular  occasion are likely similar  to those  which would have been obtained if it had been administered  to the same students with the same ability, but a different time. The more  similar the scores would have been, the more reliable the test is said to be (Hughes, 2003).

 

6.      Estimating the validity of the test in term of content validity through analyzing the CS/BC covered in the test

 A test is said to be valid if it measures accurately what it is intended to measure (Hughes,2003). Does  the test measure what it is intended to measure? The test was intended to measure the Basic Competences in the second semester, and the test as it is mentioned earlier, represented the Basic Competences in the second semester, so it can be said that the test is valid (Heaton,1995;Hughes,2003;Gronlund,1985; Hatch and Lazaraton, 1991).