Let's open those books!

A long-standing criticism of the Singaporean educational system, particularly at the primary and secondary school levels, is the prevalence of rote learning (a.k.a. "mugging" in Singaporean vernacular). Students memorize tonnes of facts and formulas and regurgitate it during their exams like so many Merlions. These students end up doing reasonably well in their standardized 'O'-level examinations, but they lack any deep understanding of the material.

This problem has been recognized by the Ministry of Education, and steps have been taken in order to address it (e.g., the Teach Less, Learn More initiative). Accordingly, one of the primary objectives of Ngee Ann Polytechnic is to produce well-rounded students with in-depth understanding of the material so that they can effectively apply them to real-life situations.

I taught two modules last semester, Object Oriented Analysis and Design (OOAD) and Computing Mathematics (CM). I observed that many students, following their tried-and-tested tactic of mugging, spent a significant amount of time memorizing facts and formulas without necessarily understanding them. For example, during study week I was called down to one of the classrooms before the OOAD exam to answer some queries by some of my students who have formed a study group. When I got there, they were in the process of reproducing the diagram for the Waterfall Software Development Life Cycle (SDLC) as given in the lecture slides:

The Waterfall SDLC. Actually, it's more accurate to say a Waterfall SDLC.

The students were memorizing the seven phases as depicted in the diagram. Not only that, they were memorizing the bullet points given in the slides that explain what each phase involves. Note that the Waterfall SDLC simply describes the method of organizing a project such that each phase is completed (together with the necessary documentation and safeguards) before the next phase begins. There is no one accepted "official" Waterfall SDLC in terms of what the actual phases are; the diagram above just happens to be the one given in our recommended textbook. Search "Waterfall SDLC" in Google Images and you'll find a dozen different depictions. Can you say, "missing the forest for the trees?"

Unfortunately, I can't blame the students. One of the questions in the exam asked for a diagrammatic explanation of the most appropriate SDLC for a case study along with explanations of the phases. Assuming the correct SDLC was identified, students who "mugged" the diagram and bullet point explanations would have scored full marks for that question. Students are nothing if not results-oriented, and mugging gets them results.

There were similar issues for the CM module. Students have to memorize and recall various formulas such as how to calculate variance and conditional probability during their exams, which distracts them from the task of understanding and applying said formulas correctly. Admittedly, people who are familiar with these formulas (such as us no-life lecturers) should be able to recall them without using a reference, but these are first-year students who have been introduced to many of these concepts for the first time. Why give them the extra burden of recall if the emphasis is on understanding?

Both of these modules have primarily closed-book assessments, and so are most of the modules in Ngee Ann. In fact, there seems to be an unwritten rule worldwide that assessments should be closed-book by default. However, for the purpose of promoting deeper understanding and lateral thinking, I am a firm believer of open-book assessments, where the students have access to reference material during the assessment. In fact, if possible I would let the students have access to the Internet, because most of them get their information from the likes of Google and Wikipedia anyway.

Let's examine some of the more common arguments for closed-book (and against open-book) assessments, shall we?

Argument #1: Sometimes you need to recall the facts instantaneously

This is a favourite argument for those who prefer to keep the reference materials away from students during exams. "Sometimes, when you are on the job, you don't have access to reference materials or you don't have time. Therefore, students should get used to not having reference materials." Frankly, I don't buy it.

Firstly, there aren't that many jobs where you can't take time out to check out your reference materials. We're living in an information age where most people bring the Internet along with them on their mobile phones or laptops, so Google and Wikipedia are just a tap or mouse-click away. Even if you're somehow unable to get an Internet connection, unless you happen to be studying how to defuse bombs, perform field surgery in a combat situation or juggle chainsaws, you can generally take the time to get to your reference materials.

Secondly, I seriously doubt that having closed-book exams prepares a student for on-the-job situations anyway. The ability to recall relevant facts comes mainly from repetition and frequent use, not from memorizing them for an exam. For example, I wrote a paper a couple of years ago where I proposed a heuristic for a rectangle packing problem, but I wouldn't be able to recall it now. On the other hand, I've been teaching UML class diagrams for the past two semesters, so I'd be pretty confident about coming up with a class diagram for any given system without using reference materials.

The point I'm making is that not allowing the students access to reference materials requires them to memorize facts that they could easily look up. This distracts them from the more important task, which is figuring out how to best apply the concepts in actual situations. Let them acquire the ability to do stuff unassisted when they're actually on the job, not when they're still learning the basics.

Argument #2: We should test students on their recall ability, which is impossible if the assessment is open-book

Closed-book exams allow recall-type questions, such as asking the student to provide a definition of a term or to reproduce a mathematical formula. The reasoning goes, "I would like to see if the student knows floozle, but if the exam is open-book, then the student can just copy from the textbook. By making it closed-book, I can ask for the definition of floozle to see if the student truly understands floozle." For a field such as law, where an accomplished lawyer should be able to quote a piece of legislation, this reasoning seems appropriate. But it's not.

There is no real benefit to being able to memorize and regurgitate, say, Section 205 Subsection 2 of the Singapore Copyright Act, not if you know how to find the relevant section if you were given the legislation document. The value of a good lawyer is in his interpretation of the law, and the same is true for other fields. Reproducing the definition of floozle does not show understanding of floozle, it shows the ability to memorize text.

So how would one test for understanding of a concept? Place the concept in a new context to see if the student is able to apply it correctly. Give him a scenario and ask him to evaluate the advantages and disadvantages of floozling as it relates to this particular scenario. This forces the student to apply the concept rather than simply recall it. Coincidentally, open-book assessments must necessarily consist of such application- or analysis-centric questions.

Argument #3: Students might spend too much time looking up information during the exam, time that would have been saved if it have been memorized

I've heard this argument before, and it's actually based on fact. I've observed that when some students have a stack of textbooks and notes with them in the exam hall, their OCD kicks in and they tend to spend an inordinate amount of time checking and re-checking their answers just to make sure that everything is correct. As a result, they might run out of time and not be able to complete the exam. 

This phenomenon is the result of the students not knowing how to best use reference materials; they are so used to mugging for closed-book exams that when the assessment is open-book, they don't know what to do. The ability to quickly find, parse, assess, analyze and apply reference material is an important skill to have, and it's a skill that many students lack. Therefore, rather than giving them closed-book exams so that they don't have to reference materials under time pressure, we should do the opposite. Let's give them a reason to acquire this skill. After all, students are nothing if not results-oriented.

Another issue with open-book assessments - financial considerations

Argument #4: Open-book exams are hard to execute

And herein lies the main reason for not having open-book assessments: in general, they're much harder to execute than closed-book assessments. For certain topics like math, this isn't such a big issue, but for other topics it's a major pain in the posterior.

When the students have access to external materials, you can no longer ask them straightforward questions like "define floozle" or "list all the types of floozle", or even "describe the advantages and disadvantages of floozles." Instead, all questions have to have some element of analysis and/or application, usually based on an example or case study. Such questions also often have multiple valid answers (that's the nature of real-life scenarios), so they're difficult to mark objectively.

Some of you may be thinking, "sure it's hard to do, but as conscientious and hardworking lecturers you should just suck it up and do it anyway." Well, this semester I'm the module leader for Advanced Object Oriented Analysis and Design (AOOAD) and I elected to make all assessments open-book. Let me tell you, creating the assessments was not easy, and marking them was even worse. There are only 33 students taking AOOAD in total, and for some of the class quizzes I received many different answers to certain questions. For each of them I had to evaluate if it was correct, and if not I still had to figure out a fair way to award partial credit. This rarely happened for the closed-book assessments of OOAD, simply due to the nature of the questions.

Extrapolate this issue over a module with larger enrolment (such as OOAD with about 150 students) and you can imagine the difficulties involved. True, a team of dedicated lecturers should be able to handle this with lots of sweat, toil and overtime, but the number of man-hours available is limited. Is it worth spending so much time creating and marking open-book assessments, or should we just use closed-book assessments and spend the time elsewhere?

In conclusion, I feel that open-book assessments should be seriously considered whenever possible. They teach students important skills such as how to use properly and efficiently use reference materials, de-emphasizes rote learning and emphasizes understanding and application. It's not easy, but if there is enough time and energy to execute it properly, I believe the benefits are substantial. The book vendors might thank you too.

How my AOOAD students addressed the financial issue

Or maybe not.