This article is based on the transcription of Matt Saunders’ presentation at the seminar ”Discussing CAN-DO statements and 4 skills assessment”, held on July 28, 2018, in Tokyo.
Our Situation At Ritsumeikan Asia Pacific University (APU) we have around 6,000 students, half of which are from Japan and the rest from overseas – from around 89 different countries. It’s quite an interesting mix of people. At APU students have either Japanese or English as their base language for classes. Those students who have Japanese as their base language have to reach a certain level in English and those with English have to reach a certain level in Japanese. So, we have quite a big language program.
In the English language program, there are 2 tracks: The Advanced track and the Standard track. Not many people get into the Advanced track where students begin at about B2 level, which is quite high, and the target is C1. Standard track students can start at zero, depending on where they come from and the target is B1+. The standard track holds about 1,200 students at any given time, and they are divided into 4 levels: Elementary, Pre-intermediate, Intermediate, and Advanced.
Our Problem As with all university programs, we needed to assign scores to students. Many of our assessments are created "in house", but we also needed a way to objectively report how our students were doing and the progress they were making by using an outside assessment. Practically, we needed something that was easy to administer, not too expensive and it had to be valid and credible. Naturally, APU wants to be able to say on their brochures that their students can get good scores on a reliable test. So, what could we do?
Thoughts on Testing and Pole Vaulters Well, before I get to how we solved our problem, let me digress a little and talk about testing in general and why it’s important. I think the key thing is that testing in general has become a universal feature of our lives. We assess people and things all the time. We have eye tests, and blood tests, and driving tests and so on. It’s not a new phenomenon – testing goes back as for as long as we’ve had social norms, back to tests of bravery or tests for coming of age.
And we’re all test makers, too. We are constantly testing people.
We are testing people in every kind of interaction that we have. So, for example, just in a conversation you could be gauging the person's knowledge on a topic, you could be testing even little things like how their English ability is whether they can handle themselves in Japanese. You’re always, always assessing these things. When you're reading you're critically thinking about what you're reading, seeing if you agree or not, and whether it’s worth investing the time to continue.
But I think that the main thing to think about is that testing is about making inferences. We test so we can infer how someone would act in a real-life situation. So, we give them a test of say, speaking, and then we think: how well will they do in a real situation where they're trying to use that same language skill? We want tests that help us (and the person being tested) understand how they will do when they get out there and have to do it…
Caring about the real-life performance is important. Imagine a test for pole vaulters. You need to run pretty fast with the pole to be a good pole vaulter. But what happens if we only focus on running speed? We have a bunch of people run really fast and we take the two fastest people and assume that they're going to be our best pole vaulters –– but we never check to see if they can get over the bar. We just check how fast they can run. In that case we end up with these great athletes and we end up recommending them to the Olympic committee and we say "Hey, these are going to be your best pole vaulters. They're the fastest two people.” but unfortunately we never did actually check if they could get over that bar. It turns out we didn't have a good measure for making an inference about this activity and what happens is if we test a limited number of skills to report our students’ overall ability it's like looking at sprinting speed to understand whether they can get over the bar…
And so, there's an important second issue here: if they know they're only being tested on their running speed they may stop trying to improve their balance or their core strength or their pole manipulation. The coaches also know all they've got to do is get this person to run fast and they’re going to be selected. Now replace these with their English equivalents. If they're only being tested on reading and grammar they're not going to work hard in the other areas. I think that the teachers, whether they like it or not, they tend to also focus on those skills because they want to see their students get a higher score and if they're only being tested on one area we're going to try and push them to do well in that area. But if you see the whole picture it means we should be testing more than just, for example, listening and reading.
Solving Our Problem Finally, back to the question. What test do we use? Well, Progress was a great solution for APU. The Progress test is a computer-based test. It tests the four skills plus grammar and vocabulary. We were using a different test that was only testing listening, grammar, vocabulary and reading but it was about two and a half hours long and we weren't getting a complete picture of our students' skills so when we switched to Progress we were able to test all the skills that we were interested in. It was cheaper. It was more thorough and it was much shorter.
The other big thing is that when we were practicing for the old tests we had huge textbooks that helped us practice for listening and grammar but concentrating on those skills took away from the time we had to practice writing and speaking. Because Progress covers more skills it means that we could actually spend time on those skills and have our administrators still be happy to see scores going up. Even though we're spending time on writing and speaking they're seeing all their scores jumping up which is really good.
Student Reaction With the Progress test, we see students are wide awake and actively involved and I think that is because there are so many different kinds of questions and so many different skills being tested. It's refreshing and wonderful to see everyone's head up and actually working on the test! Students themselves actually see the value in it: I asked them to write a letter to their kohai and give them advice, "What should I do on Progress? How can I be successful?" They had a lot of advice and they wrote more than I expected, but if I can only share one thing it would be that most students wrote:” Study hard."
We tell our students, "Study hard!” all the time. But students don't tell each other that very often and I think that's a really good indicator to me that this is the kind of test that they feel motivated to take and they know, if they study hard, it will make a difference. So, I think that was one of the really nice things about Progress, a real positive change in motivation.
Conclusion At APU scores are going up. Progress is the kind of test that if you test too often, too close together, you may not see big jumps, but over a semester, over two semesters, we're seeing lots of gains and I believe you will, too.