January 2009 Archives
Pay the People What They are Worth
By Dr. Edwin Weaver
January 26, 2009
We all want to be paid for all the work we do and when we are not paid, we feel cheated. When that happens, usually, we will avoid that type of work in the future. The same hold true in education.
What does your scoring say to the students? How you weight quizzes and exams marks sends a strong message to your students. You may think you are being fair and equal about everything, but is that how the student sees it.
At the university where I worked teaching English, the program director wanted the four main language skills weighted the same (reading, writing, language knowledge and listening). This sounded fair and correct, however the way that the university assessed these four skills were different.
To assess the language knowledge (vocabulary) the students received multiple-choice questions, outside of memory, not much work required. To assess listening, the student listens to a taped conversation and answered mostly multiple-choice and true/false questions. Even the assessment of the reading skill used multiple-choice questions.
Now, the writing is entirely different. The students had to write a 500-word essay. The essay was based on a question, which was related to the reading and listening component. Connecting the writing task to the listen and reading is great however, the students must write 500 words or more. This requires much more work than answering multiple-choice and true/false questions, not to mention you are incorporating the listening task and reading task.
Each section of the exam received the same amount of marks. The exam consisted of 20 questions from language knowledge, the 25 questions from reading, the 20 questions from listening, and the 500-word essay (with an outline). Each section, Language Knowledge, Listening, Reading and Writing all received the same amount of marks; 25% of the value of the mid-term or final exam (the final exam for Foundation year was 50% of their final grade and the mid-term was 25% of the final grade).
Here is the question to my fellow educators. How do you suppose this affects the students? Knowing that the amount of work required for writing an essay has no more value than answering multiple-choice and true/false questions, where would you devote you efforts.
Of course, the students concentrate on the first three sections and if time permits, they go to the writing section and write something. I have invigilated many of these exams and watched the students ignore the writing section and concentrate on the solving the multiple-choice questions. Afterwards they would spend 10 to 20 minutes writing something, often not even answering the question.
Their actions made a clear declaration that they did not value the writing section because it was not worth it. I followed up my theory by giving the students of my classes and two other classes a survey. The results showed that the students felt that 1) the writing took too much time, 2) they could score higher by focusing on the multiple-choice questions, and most importantly that 3) they would attempt to write the essay if it was worth more on the exam.
This attitude also affected their learning. Attempting to teach these students how to write well was very difficult. They did not see a need for it. They felt they could pass the course without learning how to write well.
Attempting to be fair, we have de-motivated the students in a particular skill. Looking at this true-life example may give you some insights on your own classes.
How do you weigh the different tasks the students perform? How do the students perceive the marking? In our case, we performed a survey to gather data on student's perception to the different tasks and the value of each of those tasks.
Assessing the impact of our assessments is very important for the continued success of our students. Take a look at how you assess your students, how you mark the assessments and the weight you assign to each task and ask yourself and your students if it is fair. You may be surprised at the answers you get.
The Good News about the Economic Crisis
By Dr. Edwin Weaver
January 23, 2009
People are worrying about their jobs and the next paycheck. Companies are closing stores and budgets cuts are happening everywhere. Banks are reeling, yet there is a bright side to this economic turmoil, especially for educators.
For years, many educators have been complaining about the standard of education. Countries around the world were pushing people through their educational system in the same manner as GM pushes cars through its assembly line. Often governments pushed people through who were not qualified in order to get people off the streets and enough warm bodies into the market. The end product was mediocre workers. That will change.
The competition for jobs will increase steadily for the next two to five years. Only the best, only the most qualified people will get jobs. Companies and governments cannot afford to spend money on bringing people up to speed or hiring two people to do the job of one. They need people who are qualified and ready to start work.
Governments will start demanding that everyone who exits an institution of higher education is 100% qualified. They will demand that their people are competitive in a global market. As the colleges and universities start turning down admission to students who are not ready to study at higher levels, governments will put pressure on the high schools to ensure the quality of their graduates.
This is where the benefit to the educator comes in. We have labored under antiquated systems. We have used meaningless curriculums, which would not produce the caliber of student, which today's world needs. We know of methods of teaching, which research has proven, but were unable to implement against the status quo. All of these things will change.
Governments and administrators will become receptive to improvements. They will implement new methods of teaching. They will listen to complaints about the weaknesses in curriculum. They will want to overhaul the educational system to produce world-class graduates. They must overhaul the system for their people to survive. They will listen.
Now is the time for educators to gather all their data, all the facts that prove a particular system does not function, as it should. Combine these facts with sound arguments, backed by research and present a plan for change. This is the hour of the educator. Do not allow this opportunity to slide by.
Is Your Assessment Valid?
By Dr. Edwin Weaver
January 2008
Recently, while working at a university, the rules changed for taking mid-term and final exams. Prior to this change each section of the exam was timed. Of course, everyone understands the importance of timed exams on producing assessments which are considered valid in the eyes of other institutes. The change was brought about due to the complaints of students that there was not enough time to complete the exam.
The original tests were designed to give the student, what is considered, the proper time to respond, provided that the student has a working knowledge of the information. To present an example; the old multiple choice section provided 1.1 minutes per question. Under the new system, there is no real time limit for the multiple choice section.
We are focusing on validity not reliability. Reliability has to do with the consistency of scoring between markers where as validity is focused on the consistency results when used in different groups of student (yet the same level), different schools, different school years, etc.
Mid-term and final exams are high stakes assessments. They determine whether a student proceeds to the next level or the next year in the university. High stakes assessments must be proven valid, especially if the university wants to be considered as a legitimate institute of education.
In order for an assessment to be considered valid there must be generalizability. In other words, "the degree to which evidence of validity based on test-criterion relations can be generalized to a new situation without further study of validity in that new situation." AERA et al (1999 p. 15). You can not get the same results in different areas because each person will dedicate a different amount of time to the task.
AERA et al (1999) report states that 'time period' or allotted time per item or section is an important aspect in measurement consistency. Without consistent measurements how can we properly evaluate the curriculum? Without consistent measurements how can we properly evaluate teacher efficiency? We use the 'timed' aspect to our exams otherwise our results will be skewed.
This skewing of results has occurred. In the latest exams, we had students who scored extremely high on the reading section, but very low on the vocabulary section and others who scored extremely high on writing, but failed the vocabulary and reading. Each student chose the area which they wanted to dedicate their time.
These skewed results make it impossible to evaluate the learning process and the educational program as a whole. It makes it impossible to find problem areas in the program or to make adjustments to the program. Exam validity is a cornerstone to our education system, no matter what the students may think.
As to the student's complaints of not enough time to produce good work, that complaint is but a myth. Kroll (1990) found that the timing really did not make a difference. Polio et al. (1998) came up with the same results. TESL did a study in 2006 and found basically the same results. They noted the only thing which will make a difference in the writing section is the use of technology, computers. Otherwise, giving the student more time to work on a task basically only helped in the spelling component.
Therefore, changing our allotted time scheme and allowing students to determine how much time they spend on each task will not permit us to have truly valid tests. We can not change to match the whims of the students. The student must learn to function under the dictates of the world around them. Yes, we should listen to their complaints and assess whether their points are valid, but to bow to the whims of the students is not acceptable.
These types of complaints are but one of the consequences of the assessment. There are consequences to every assessment; both good and bad consequence. That is part of assessment development. The consequence of students complaining about time restraint is to be expected and as AERA et al (1999) says these consequences do not invalidate the assessment.
If we are to assess the learning of our students, if we are to assess the programs we use to teach the students, all exams must be valid. The must produce accurate data to ensure the success of the students and the institution. If you are coming under fire about exams and are thinking about changing the exams, think twice. You may not be helping the students or the institution, you may be leading it into disaster.
References:
American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME). (1999) Standards for educational and psychological testing.
Kroll, B. (1990). What does time buy? ESL student performance on home versus class compositions. In B. Kroll (Ed.), Second language writing: Research insights from the classroom (pp. 140-154).
Polio, C., Fleck, C., & Leder, N. (1998). "If I only had more time: ESL learners' changes in linguistic accuracy on essay revisions." Journal of Second Language Writing, 7(1), 43-68.
