About Innovative Question Papers

The typical exams, at least in Indian universities, tend to be really dull, drab affairs. I cannot speak about how good (or bad) they are in other countries, but partly by design, partly due to inertia, they follow the same old template in most Indian universities.

You’ll get your fair share of “State briefly”, a sprinkling of “Explain why”, and the very occasional “In your opinion”. But for the most part, it needs rote memorization to respond and “do well” in these examinations. Take home essays, reflective essays and anything that is even remotely innovative are not welcome.

But in a world where such a thing was possible, what might question papers look like?

Tyler [Cowen] once walked into class the day of the final exam and he said. “Here is the exam. Write your own questions. Write your own answers. Harder questions and better answers get more points.” Then he walked out. The funniest thing was when a student came in late and I had to explain to him what the exam was and he didn’t believe me!

https://chrisblattman.com/2012/08/15/best-exam-question-ever/
https://chrisblattman.com/files/2013/01/1-Introduction.pdf

I can’t find it online right now, but I seem to remember that some of Prof. Blattman’s papers included the additional condition that you needed to submit the cheat sheet, and it too would be evaluated for what you chose to put in it (and implicitly, what you chose to leave out).


I had once asked a very small development economics class to teach some of their peers – those who hadn’t taken Dev Eco – the Solow Model. The non-math version, of course. And then those who had been taught had to take a small multiple choice question test. The marks they scored would become the grades for the “teachers”. Stuff like this will never work for a larger class, will never work for a semester end exam, but it still was a lot of fun to do!

The point is that it is all too possible to have examinations be fun, and be about learning. But so long as examinations are about signaling how much you’ve scored, rather than allowing you to reflect on how much you’ve learnt, they’ll have to be standardized, which means they’ll have to continue to be boring.

If you’d like them to be fun, you need to try and live in a world where examinations (and therefore colleges) aren’t about signaling.


But that is a tall order, no?

Proposed Examination Reforms

I’m not holding my breath, but this article has raised my hopes just a little bit:

Colleges and universities may soon adopt continuous comprehensive evaluation, a method that shifts focus from only annual or semester-level summative assessment system.
The suggestion has come from higher education regulator University Grants Commission (UGC) amid the increasing dependence on technology for education delivery in the current pandemic environment. Assessment at several intervals during and after achievement of learning outcomes specified for every module is needed as blended learning is gaining ground, UGC said in a draft proposal shared with higher educational institutions.

https://www.livemint.com/education/news/ugc-bats-for-reforms-in-exams-with-a-focus-on-continuous-evaluation-11621798197326.html

I’ve said it before, and I’ll say it again: everybody associated with academia in India knows how broken, pointless and screwed-up examinations are, but nobody wants to do anything about it. And the most often quoted reason is R Madhavan in 3 Idiots going “Abba nahi manenge”.

Abba being the UGC.

But now, hallelujah, the UGC is talking about open book examinations and on-demand examinations. This was a “tears in my eyes moment“:

Open book exams is the “right way to move away from the conventional approach of exam where remembering and reproducing is prime”, UGC said. “In real functioning beyond formal education, life is all about open book examination. Hence, in higher education, we must prepare students for work life by making them acquainted with open book examinations. It will also facilitate better understanding and application of knowledge,” UGC said, citing an internal committee report.

https://www.livemint.com/education/news/ugc-bats-for-reforms-in-exams-with-a-focus-on-continuous-evaluation-11621798197326.html

There is still a world of pain that awaits those of us who are in academia. The inertia associated with the old system will take years1 to overcome, and it will be a long, unpleasant journey.

More, we will run up against capacity constraints, because shifting away from the “State in brief” questions to having students think critically will require the changing of multiple mindsets, along with intensive training of faculty in all universities.

And even if some universities were to adopt this whole-heartedly, one unintended consequence will be the exacerbation of already ridiculously high inequality. The inequality I speak of is in terms of access to quality higher education, of course. Better colleges and universities will get better still, and while that is desirable for the students who are lucky enough to get into them, it doesn’t bode well for equitable educational outcomes across the country.

But even so, the very fact that this is even being discussed in the first place is a welcome move.


The one thing that gives me hope is something that is discussed almost as an after-thought in the article: e-portfolios.

An electronic porfolio (e-portfolio) is a purposeful collection of sample student work, demonstrations, and artifacts that showcase student’s learning progression, achievement, and evidence of what students can do. The collection can include essays and papers (text-based), blog, multimedia (recordings of demonstrations, interviews, presentations, etc.), graphic.

https://teaching.berkeley.edu/resources/assessment-and-evaluation/design-assessment/e-portfolio

This blog, for example, is my “e-portfolio”. I pay around ten to twelve thousand rupees every year to maintain this blog, but one can of course start a blog for free. Or a YouTube channel, or an Instagram page or absolutely anything else you like.

In an ideal world, e-portfolios (and could we come up with e better name for it, please?) are solely the responsibility of the student. They can be in any language. They can be nurtured over time, for years together. Cultivating your e-portfolio needn’t cost money, in other words, and popularizing your e-portfolio is a life-skill worth developing in its own right.

Most importantly, developing one requires just a smartphone. Yes, this is still a challenge for large parts of our country, but I would argue that a learning system that revolves around the development of an e-portfolio is more efficient, cheaper and easier than even a perfectly reformed examination system.


Bottomline: marks, examinations and degrees are overrated. Doing the work, and sharing your work in the public domain is underrated.


Here’s a blogpost from last year about conducting examinations during these crazy times, and here are all the posts I have written about higher education on EFE.

  1. And if I am to be cynical, which is almost always the case, I’ll say decades[]

On Conducting Examinations, Especially in 2020

This is less a blogpost, and more of a rant. Consider yourself warned! 🙂

This past Saturday, I cam across a most excellent Twitter thread:

The author, Carl Bergstrom (Wikipedia article here, University profile here) makes a detailed, reasoned argument against online proctoring of examinations, especially in 2020. In my blogpost, I’m going to riff on this Twitter thread, and some related points, and build an argument against the way we conduct examinations in Indian Universities.

To begin, watch this Sugata Mitra TED talk about education:

Just the first five minutes or so is enough for the argument I will be making today, but you really should watch the full thing. There was, as it turns out, a case for rote memorization at one point of time. But today, as Mitra says in the video, it is the computers that are the clerks. They do (and to be fair to them, they do it much better than we ever could) the job of remembering everything, so that we don’t have to.

To every single professor reading this blogpost: when was the last time you yourself spent a day working without looking up something on the internet? When was the last time you researched something, wrote something, without using your computer, or the internet on some device?

Then why do we insist on examining our students for their ability to do so, when we ourselves don’t do it? And for those of our students who are not going to get into academia, they wouldn’t last in their organizations for even an hour if they tried to work without using computers and the internet.

They would (should!) be fired for being Luddites!

And yet, to land up in that firm, they must spend the last week of their lives as a college student cooped up for three hours in a classroom without a computer, without the internet, and use pen and paper to write out the important features of xyz in abc lines.

I can’t possibly be the only one that spots the incongruity, surely?

Here, from the Twitter thread I spoke about earlier, is a picture of Bloom’s taxonomy:

Image

Read the Wikipedia article about Bloom’s taxonomy, or look carefully at the picture above. At best – and I think I am being charitable here – our question papers in higher education in India reach the evaluate stage, but certainly not create. And even that is a stretch.

Moreover, even if we somehow agree that we do indeed reach the evaluate stage, we are effectively asking students to evaluate based on their memory alone. Why?

One, won’t students write a better evaluation of whatever theory we are asking them to write about if we give them the ability to research while writing? Second – and I know this is repetitive, but still – are they ever going to write an evaluation without having access to to the internet?

In plain simple English: We train students for 25 years to get awesome at memorizing stuff, and then expect them to do well in a world which doesn’t value this skill at all.

(To be clear, some things you should remember, of course. Think of it as a spectrum – and I am not suggesting that we move to the end of the spectrum where no memorization is required. I am suggesting, however, that we are at the end of the spectrum where only memorization is required. Close enough, at any rate).

Coming specifically to this year, the year of online examinations, here’s a tweet that was quoted in Bergstrom’s thread:

There really isn’t much to say, is there? All universities the world over have sent out variations of this nightmare this year, and in some cases, repeatedly. It’s the whole null hypothesis argument all over again – we assume all students to be guilty until proven otherwise. That is, we assume everybody will cheat, and therefore force everybody to comply with ridiculously onerous rules – so as to prevent the few who might actually cheat.

And cheating, of course, being looking up stuff on the internet. The argument itself is pointless, as I have explained above, and we go to eye-popping lengths to enforce the logical outcome of this pointless argument.

Prof. Bergstrom makes the same points himself in the Twitter thread, of course:

This year, especially, is a good opportunity to turn what is otherwise a disaster of a situation into meaningful reform of the way we conduct examinations.

Students, parents – indeed society at large – will spot the incongruity of learning online, but being examined offline. If we, in higher education in India do not spot this incongruity and work towards changing it – well then, we will have failed.

And finally, the last tweet in the thread is something we would all do well to remember:

Links about these changing times

  1. Agnes Callard on wanting to feel pain:
    ..
    ..
    “We don’t consciously choose to feel pangs of guilt or waves of regret, in the way that we consciously choose what novel to read. Still, we can assimilate the two sorts of cases if we introduce a hypothetical: Imagine you are offered a pill that would make you immune to regretful or guilty thoughts. Would you choose take it? If your worry is that those thoughts are important for steering you away from future wrongdoing, let me assure you I’ve built that functionality into the pill: You won’t behave any worse for having taken it. You’ll just stop having negative feelings.”
    ..
    ..
  2. Via MR, social distancing and examinations in South Korea. (I’m not a fan)
    ..
    ..
  3. Scott Adams has been calling it correctly for a while. Read more Dilbert!
    ..
    ..
  4. Again via MR, a lovely list.
    ..
    ..
  5. What will change, culturally speaking? Telecommuting will be the default, and in the years to come, maybe you’ll have to ask the manager for permission to go to office.
    ..
    ..
    “But the pandemic is forcing these investments in industries where telework is possible, with more people learning how to use remote technology. As a result, we may see a more permanent shift toward telecommuting. As the economist Susan Athey recently told the Washington Post, “People will change their habits, and some of these habits will stick. There’s a lot of things where people are just slowly shifting, and this will accelerate that.””

Complements, Substitutes and Examinations

Writing all of what I wrote in February 2020 was a lot of fun, and gave rise to a series of interesting, and interlinked ideas.

In today’s essay, I want to explore one of these interlinked ideas: I want to riff on the concept made famous by Steve Jobs: the computer as a bicycle for the mind. But with an Econ 101 twist to the topic!

I’ve already linked to the video where Steve Jobs speaks about this, but just in case you haven’t seen it, here’s the video:

As I mentioned in the post “Apple Through Five Articles“, Steve Jobs was essentially saying that the computer is a complementary good for the mind: that the mind becomes far more powerful, far more useful as a tool when used in conjunction with a computer.

A complement refers to a complementary good or service used in conjunction with another good or service. Usually, the complementary good has little to no value when consumed alone, but when combined with another good or service, it adds to the overall value of the offering. A product can be considered a compliment (sic) when it shares a beneficial relationship with another product offering, for example, an iPhone complements an app.

One way to understand Apple is to understand that Jobs effectively ensured that Apple built better and better computers. Apple has continued to do that even after Jobs has passed on, but they’ve been building computers all along. You can call them Macs and iPhones and iPads and Apple Watches, but they’re really computers.

But that’s not the focus of this piece. The focus of this piece is to think about this as an economist. If the mind is made more useful when it is able to complement the processing power of the computer, then the world is obviously more productive now that many more minds are being complemented with many more computers. I writing this piece on my laptop, and you reading it on your device is the most appropriate example – or so we shall assume.

But viewed this way, I would argue that we get the design of most of our examinations wrong. Rote memorization, or “mugging up” is still the default method for evaluating whether a student has learnt a particular subject. Mugging up is just another way of saying that we need to substitute for the computer, not complement it!

When we reject open book examinations, when we reject the ability to write a paper using laptops/tablets that are connected to the internet, when we force students to substitute for computers, rather than use them to write better, richer, more informed answers, we’re actively rejecting the analogy of the bicycle for the mind.

To say nothing, of course, of the irrelevance of forcing people to write examinations for three hours using pen and paper. But that’s a topic for another day.

Right now, suffice it to say that when it comes to examinations in India, Steve Jobs would almost certainly have not approved.

Bottom line: If computers are a complement, our examinations are incorrectly designed, and we end up testing skills that are no longer relevant.

And the meta-skill you might take away from this essay is the fact that a lot of ideas in economics are applicable in entirely surprising and unexpected areas!

I hope some of you disagree, and we can argue a bit about this. I look forward to it! 🙂