AStudyofValidityandReliabilityofContent—basedWritingbyEnglishMajorStudents

單俊林+王丹

【Abstract】In nature， writing is a linguistic， social-cultural and cognitive activity. Through assessment， students practical language competence could be reflected， what can also be reflected are the teachers success or teaching deficiency. This study takes assignments from two undergraduate classes， the contents and scores of the students writings are analyzed by reliability and validity， with the aim of obtaining a feedback which would be helpful to future teaching and learning.

【Key words】reliability； validity； writing

1. Introduction

Effective language test measures how many teaching goals have been achieved， how much knowledge students have learned in order to get positive washback effects of the instruction implemented. Writing is regarded as important aspect of language output. Writing test is valued as one of the most scientific and effective ways to evaluate learners language proficiency， and thus becomes an important and necessary component of many large-scale English tests.

2. Students writing test

Diverse types of language test exist and they differ from one another due to the test method and purpose. They can be classified into different perspectives， but the following four types are most common：proficiency test， diagnostic test， placement test and achievement test.

This paper is based on achievement tests taken by two English Major classes. In the test， students are asked to write compositions based on short stories from the textbook， and the composition should contain two parts：a short review and a literature criticism. The length of the composition is not strictly limited and the number of students is around 40.

2.1 Achievement test

Achievement test is generally divided into two types：final achievement test and progress achievement test. As the terms indicate， “final achievement test is conducted at the end of a study course”；while progress achievement test “measure the progress that students are making”. Undoubtedly， the content of these tests must be relevant to the given courses.

Different from proficiency tests， achievement tests have something to do with language courses， and the purpose of them are to assess “how successful an individual student， groups of students， or the courses themselves have been in achieving the objectives”.

2.2 Writing test

Writing test is used when the teacher wants to know whether the students have mastered what has been taught in class according to certain teaching objectives. An achievement test is related directly to the lessons， or even a curriculum. Therefore， a writing test should be narrowly focused on particular teaching and learning goals and curriculum， and hence it is an achievement test.

Teachers tailor writing tasks by the goals of the course and the backgrounds and needs of the students， and build in both authenticity and interactivity into the writing test tasks. So， the writing assessment should be as closely related as possible to writing instruction and teaching curriculum so as to provide information for teachers to evaluate their students progress and abilities in writing.

2.3 Content-based writing

This kind of writing can be regarded as an achievement test， only that students make their compositions after class. Test takers（students） are required to write on the basis of given material.

3. Reliability and Validity of the writing content

Content is the most important element in the whole process， for if anyone cheats and gets high score， the reliability would be very low. So， we have to analyze it first to make sure the reliability is high， if not， the following analysis would be in vail， because the foundation of the analysis is based on honest writing.

3.1 The validity of traditional scoring

It is the fact that the whole writing process is finished after class， both teacher and the scorers cant monitor the students and could never know if any of the students finish their assignments by copying others. But， according to the nature of the writing， the first part of which is brief introduction of the given short story， some students would use words or sentences from it， and if this kind of act has caused some resemblances in some writings， it would not be counted as copying， although they use them without quoting marks.

3.2 The validity of computer-/internet-assisted scoring

There are at least three ways of computer-/ internet-assisted scoring：（1） hand in the assignment onto one particular website， in which only scorers have the access to the scoring end；（2） students e-mail the assignments to a particular address respectively；（3） students hand in assignments to the monitor in digital form， then the monitor sends them together to scorers e-mail address. All the three ways above have one advantage：it can avoid bad handwritings， so as to increase scorer validity and student reliability. But， at the same time， it will decrease student validity. So， it leaves to the teacher to balance.

In internet-assisted scoring， scorers sign in the teachers account and give scores respectively and manually. Some advantages are obvious：if any student copies others， the system would point it out by warning the assignment is dubious， so as to get a higher scorer validity. However， the disadvantage is also obvious：the writings are based on reading of given short story， so the contents are restricted and tend to be similar to one another. In this condition， although students do not copy others， the system would still view some of them as copied， for the reason that the computer system only count the same words used， but not regard the content as a whole. Another disadvantage would also occur when the total student number of the two classes is only around 40. All of the students make the composition on the computer， and the scorers give scores with the assistance of computer， all these together would take longer time than traditional form of handing the assignments in. So， take the small number of students into account， computer-/ internet-assisted scoring has low validity.

4. The relevance between handwriting and score

Handwriting do have influence in the scoring process， and sometimes scorers wont examine the content of the composition and just give a very low score directly after seeing a very bad handwriting. So， handwriting is also a determinant of the final score.

4.1 Students handwriting and score

As students make their compositions under a pressure-free condition， they can bring into play their true language competence. However， they may show little care on their handwriting， spelling， grammar control， etc.， all these would lead to mistakes and score loss. Scorers cannot identify students words and would take scores off， this would be one of the most important reasons for students losing score.

4.2 Scorer Reliability and score

A phenomenon exists that the handwriting of any student is too bad to read. In fact， during the process of scoring， we can see that in some of the compositions with bad handwritings， the words are correct and their grammars have no obvious mistakes. So， conclusions could not be drawn by intuition that a student who has bad handwriting would also have bad grammar control. In this point， in the process of scoring， we have to make sure that the scorers do have patience and checked every piece of composition in detail.

4.3 Summary

Ill characters caused by bad handwriting cant be judged as occurred by chance or by high frequency. In this case scorer reliability would be affected by their patience and prudence. The more patient and prudent the scorers are in the process of scoring， the higher reliability the scores have.

5. The relevance between writing content and score

Although the handwriting is very good， it cant be counted as a good piece of composition， the content must be good. A cohesive and coherent content is what makes a high score. So， students should focus on the content before the handwriting. Of course， if a composition has both good content and good handwriting， it would be perfect.

5.1 The length of the content and score

The content of the composition is restricted， normally be a literature criticism on a short story in the textbook. There is no limited number of words， but two parts are to be included：brief review of the story and personal view. Each time students hand in their assignments on the same story， so that it is available to compare the scores between the students.

However， students writings differ in number of words. Some write a little more than 100， and some write as long as 300 to 400. If they make mistakes in the same rate， the student who write fewer words would score higher than the one who write a lot more. Also， the score is given by taking off 1 point from 1mistake each， that doesnt take cohesiveness and cohesion of the content into account. So， in the perspective of writing ability， the final scores have low reliability.

5.2 The length of the content and time limit

Students can take as much time as they would like to finish the assignment. Normally there are 3 to 4 days to write one piece of composition before handing in， and if the time for other homework is taken off， there would not be too much left. So， the time for an off-class composition has length similar of an in-class one.

5.3 Summary

Restriction on number of words is highly recommended， for example：at least 150 words should be written in the composition， while 200 words would be an upper limit. This would increase both scorer efficiency and scorer reliability. Time limitation， if students can set for themselves for a particular composition， would be welcomed. Then the writing would share many features with in-class language test， and the only difference would be that students work in a natural way which could reflect their true language competence.

6. Backwash effects on students learning

In a certain degree， all language tests are designed with a purpose of positive feedback. According to it， the teacher would get a clearer view on future teaching， make adjustments in both teaching content and method. As to students， they can know whether they have grasped the knowledge well enough and whether they need some instructions on the learning method.

6.1 Analysis from scorers perspectives

As analyzed above， scorers have to pay more attention to the scoring process， because handwriting has low relevance to language competence. The composition takes grammar and spelling as the main focus， so， bad handwriting cant be regarded as an excuse to take off points.

6.2 Analysis from students perspectives

Scorer will underline the mistakes found in students assignments and would not correct them， this work would leave to students themselves. They have various methods：check， discuss， look into dictionary， etc. Through this act， students can learn more actively and automatically， and can increase their ability of language understanding and language competence.

A phenomenon would occur for the lack of supervision：students would not review their assignments or correct their mistakes， they would rather write new assignments directly on the blank paper instead. So， whether there is a beneficial backwash depends on the students own attitude toward college life.

6.3 Analysis from teachers perspective

Through the writing assignments， the teacher would have a general view about how the students behave in their language use， and make adjustments in his/her future teaching. More effective teaching method would be developed to make the students better understand English as a language.

6.4 Summary

Underlining the mistakes rather than correcting them have more advantages than disadvantages. As to those disadvantages， one method may solve it：at the first 5 minutes of the class， desk-mates check for each others assignments and help them correct the mistakes.

7. Conclusion

Through empirical study of the scoring of assignments handed in from two undergraduate classes， we can see that content-based writing has a positive effect in the formation of language competence for English major students， although the present assignment form has disadvantages and needs adjustment. However， we cant deny that it can reflect students learning model and language competence under natural condition. More detailed empirical studies are still to be done， and data should be collected and analyzed for better instructing teacher and student in future.

References：

[1]姜蕾.大學英語課堂環(huán)境下讀寫結合寫作測試效度研究， 2011.

[2]Brown，D.Teaching by principles：An interactive approach to language pedagogy.Englewood Cliffs，NJ：Prentice Hall Regents.1994.

*本文是課題““一帶一路”背景下大學生涉外語用能力提升策略研究”的部分研究成果，課題編號：ZGWYJYJJ2016Z14。

作者簡介：

單俊林（1987-），男，黑龍江大慶人，東北林業(yè)大學外國語學院英語語言文學碩士研究生，研究方向：語用學、認知語言學。

王丹（1967-），女，黑龍江哈爾濱人，教授，碩士生導師。從事語用學、認知語言學研究。