Is Your Assessment Valid?

Is Your Assessment Valid?

By Dr. Edwin Weaver

www.we-develop-leaders.com

www.edu-talk.info

January 2008

Recently, while working at a university, the rules changed for taking mid-term and final exams. Prior to this change each section of the exam was timed. Of course, everyone understands the importance of timed exams on producing assessments which are considered valid in the eyes of other institutes. The change was brought about due to the complaints of students that there was not enough time to complete the exam.

The original tests were designed to give the student, what is considered, the proper time to respond, provided that the student has a working knowledge of the information. To present an example; the old multiple choice section provided 1.1 minutes per question. Under the new system, there is no real time limit for the multiple choice section.

We are focusing on validity not reliability. Reliability has to do with the consistency of scoring between markers where as validity is focused on the consistency results when used in different groups of student (yet the same level), different schools, different school years, etc.

Mid-term and final exams are high stakes assessments. They determine whether a student proceeds to the next level or the next year in the university. High stakes assessments must be proven valid, especially if the university wants to be considered as a legitimate institute of education.

In order for an assessment to be considered valid there must be generalizability. In other words, "the degree to which evidence of validity based on test-criterion relations can be generalized to a new situation without further study of validity in that new situation." AERA et al (1999 p. 15). You can not get the same results in different areas because each person will dedicate a different amount of time to the task.

AERA et al (1999) report states that 'time period' or allotted time per item or section is an important aspect in measurement consistency. Without consistent measurements how can we properly evaluate the curriculum? Without consistent measurements how can we properly evaluate teacher efficiency? We use the 'timed' aspect to our exams otherwise our results will be skewed.

This skewing of results has occurred. In the latest exams, we had students who scored extremely high on the reading section, but very low on the vocabulary section and others who scored extremely high on writing, but failed the vocabulary and reading. Each student chose the area which they wanted to dedicate their time.

These skewed results make it impossible to evaluate the learning process and the educational program as a whole. It makes it impossible to find problem areas in the program or to make adjustments to the program. Exam validity is a cornerstone to our education system, no matter what the students may think.

As to the student's complaints of not enough time to produce good work, that complaint is but a myth. Kroll (1990) found that the timing really did not make a difference. Polio et al. (1998) came up with the same results. TESL did a study in 2006 and found basically the same results. They noted the only thing which will make a difference in the writing section is the use of technology, computers. Otherwise, giving the student more time to work on a task basically only helped in the spelling component.

Therefore, changing our allotted time scheme and allowing students to determine how much time they spend on each task will not permit us to have truly valid tests. We can not change to match the whims of the students. The student must learn to function under the dictates of the world around them. Yes, we should listen to their complaints and assess whether their points are valid, but to bow to the whims of the students is not acceptable.

These types of complaints are but one of the consequences of the assessment. There are consequences to every assessment; both good and bad consequence. That is part of assessment development. The consequence of students complaining about time restraint is to be expected and as AERA et al (1999) says these consequences do not invalidate the assessment.

If we are to assess the learning of our students, if we are to assess the programs we use to teach the students, all exams must be valid. The must produce accurate data to ensure the success of the students and the institution. If you are coming under fire about exams and are thinking about changing the exams, think twice. You may not be helping the students or the institution, you may be leading it into disaster.

References:

American Educational Research Association (AERA), American Psychological  Association (APA), & National Council on Measurement in Education (NCME). (1999) Standards for educational and psychological testing. Washington, DC. Author.

Kroll, B. (1990). What does time buy? ESL student performance on home versus class compositions. In B. Kroll (Ed.), Second language writing: Research insights from the classroom (pp. 140-154). New York: Cambridge University Press.

Polio, C., Fleck, C., & Leder, N. (1998). "If I only had more time: ESL learners' changes in linguistic accuracy on essay revisions." Journal of Second Language Writing, 7(1), 43-68.