Tag: examinations

Professor Harberger’s Midterm and Final Examinations (and ChatGPT)

A book that hardly anybody reads these days (but really should) is The Theory of Price, by George Stigler.

Take, for example, Stigler’s “A Note on Block Booking.” Block booking of movies was the offer of a fixed package of movies to an exhibitor; the exhibitor could not pick and choose among the movies in the package. The Supreme Court banned the practice on the grounds that the movie companies were compounding a monopoly by using the popularity of the winning movies to compel exhibitors to purchase the losers.
Stigler disagreed and presented a simple alternative argument. If Gone with the Wind is worth $10,000 to the exhibitor and Getting Gertie’s Garter is worth nothing, wrote Stigler, the distributor could get the whole $10,000 by selling Gone with the Wind. Throwing in a worthless movie would not cause the exhibitor to pay any more than $10,000. Therefore, reasoned Stigler, the Supreme Court’s explanation seemed wrong.
But why did block booking exist? Stigler’s explanation was that if exhibitors valued films differently from one another, the distributor could collect more by “bundling” the movies. Stigler gave an example in which exhibitor A is willing to pay $8,000 for movie X and $2,500 for Y, and B is willing to pay $7,000 for X and $3,000 for Y. If the distributor charges a single price for each movie, his profit-maximizing price is $7,000 for X and $2,500 for Y. The distributor will then collect $9,500 each from A and B, for a total of $19,000. But with block booking the seller can charge $10,000 (A and B each value the two movies combined at $10,000 or more) for the bundle and make $20,000. Stigler then went on to suggest some empirical tests of his argument and actually did one, showing that customers’ relative tastes for movies, as measured by box office receipts, did differ from city to city.

https://www.econlib.org/library/Enc/bios/Stigler.html

If you want a more “modern” take, you could read this lovely essay, by Chris Dixon.

But to come back to Stigler, one reason to read his book is because “a typical Stigler article laid out a new proposition with clear reasoning and then presented simple but persuasive data to back up his argument”. Which, if you ask me, is a better way to teach microeconomics than via real analysis.

If you aren’t yet subscribed to Irwin Collier’s blog, please do so. His blog is a delightful treasure trove of question papers from the past, and some of these papers really and truly make you think. Here, for example is just one question from November 5, 1957. This particular question paper was set by Professor Harberger (and the examination took place a year before Stigler joined the University of Chicago), but nonetheless, it is a great way to check how well you know price theory microeconomics.

The price elasticity of demand for a good will be higher, the higher is the income elasticity of demand for that good.

For a question like this (and before you run it through ChatGPT), think about whether the statement makes intuitive sense. If the income elasticity of demand for a good is high, that means that small changes in income will lead to large changes in demand for that good. Eating out at expensive restaurants will go down if you lose your job, for example.

Well, then, yes, you might think – the price elasticity also ought to be high then, no?

So it would seem that this is true then.

Ah, but is it? Do you know of folks who buy the latest iPhone regardless of how much (and whether) they earn? Can you, in other words, think of a single counter-example to your own argument? Does your answer change? If yes, why? If not, why not?

(My own take is that this is false, by the way. Do you agree?)

Now that you have thought a little about this yourself, ask ChatGPT what it thinks (or Bard, etc.). See if your answers match. If they don’t, have a conversation with it. Tell it why you thought your answer was whatever it was, and ask it to critique your answer.

Rinse and repeat for all the other questions in the examination, and I envy you your journey of learning price theory. Oh, and by the way, if you don’t like your current (ahem) price theory Prof, feel free to call upon the knowledge and abilities of Professor Stigler himself:

“What book should I read to learn economics?”

“The Internet, and ChatGPT!”

Learn at Twitter Speed, Get Tested at AOL Speed

The title of today’s post is directly lifted from an MR Post from yesterday, which you should read in its entirety.

Instead of hearing a rumor at the coffee shop and running down to the bank branch to wait on line to withdraw your money, now you can hear a rumor on Twitter or the group chat and use an app to withdraw money instantly. A tech-friendly bank with a highly digitally connected set of depositors can lose 25% of its deposits in hours, which did not seem conceivable in previous eras of bank runs.
But the other part of the problem is that, while depositors can panic faster and banks can give them their money faster, the lender-of-last-resort system on which all of this relies is still stuck in a slower, more leisurely era. “When the user interface improves faster than the core system, it means customers can act faster than the bank can react,” wrote Byrne Hobart. You can panic in an instant and withdraw your money with an app, but the bank can’t get more money without a series of phone calls and test trades that can only happen during regular business hours.

https://marginalrevolution.com/marginalrevolution/2023/03/banks-as-meme-stocks.html

Try this variant on for size:

Instead of hearing about a concept in a classroom, and running to the library to get access to the book that explains it in greater detail, now you can hear about a concept on Twitter, or the group chat, and use ChatGPT to learn all about it instantly. A tech-friendly classroom with a highly digitally connected group of learners can learn much more about a topic in a couple of hours, which did not seem conceivable in previous learning environments.
But the other part of the problem is that, while learners can learn faster and LLM’s can give them additional nuance and context much better, the exam system on which all of this ultimately relies for certifications is still stuck in a slower, more traditional era. “When the learning environment improves faster than the testing environment, it means learners can learn better than colleges can meaningfully test them,” wrote a grumpy old blogger. You can learn much more about a topic in a semester than you ever could before, but the college will still insist on making you memorize stuff so that you can choose five questions out of six to answer in a closed-book-pen-and-paper examination.

It’s not an exact analogy, of course. But there are two points to this blogpost:

  1. Where colleges and universities are concerned, this is a useful framework to deploy. And sure I had fun tweaking that excerpt in order to maximize my snarkiness – but I’m not joking about the point being made. When students are able to learn far better, far more effectively and far faster, but the testing environment doesn’t keep up with either the learning or its applications, it is a problem. Simply put, if teaching and learning with LLM’s is best, but the college thinks that testing without access to LLM’s is best, there’s a disconnect.
  2. The broader point, of course, is that you should be applying this framework to everything. Banks and colleges, sure. What about government (at all levels)? What about software companies? What about delivery apps? What about <insert the place you work at here>? Which parts of your organization are already using LLM’s in their workflows, or will sooner rather than later? Which parts will be the most reluctant, and therefore the last to adopt to this brave new world? What imbalances might result? How should we incentivize the rate of adoption such that we optimize appropriately?

Note that this doesn’t necessarily mean incentivizing those reluctant to adopt! You might want to incentivize a slower adoption of ChatGPT, if that’s what you think is best (and yes, that goes for colleges too). But if that’s the route you’re going to go down, think first about the competition. And note that in the age of LLM’s, defining who your competition is isn’t as easy as it used to be.

Syllabus Bleg

I’m teaching a course on behavioral economics at the Gokhale Institute this semester, and as you might imagine, it is a whole lot of fun. For a whole variety of reasons, and that’s a separate post in and of itself. But for now, I’ll tell you this much – the reason I have such a lot of fun is because teaching behavioral economics forces you to take stock of what you know to be true – before, during and after this semester is over.

And part of the reason teaching this course is so much fun is because you might think that when I speak about “know to be true”, I’m talking about ‘attacking’ the fundamental axioms of economic theory. Well, yes, but that would be a very one-sided class. What are the strongest arguments of the other side is a question that seems to have fallen out of favor in the world today – but one of the meta-lessons I hope to impart is just this one. That thinking along these lines is an important, and nowadays-forgotten skill.

Learn to ask how you might be wrong about whatever it is you think you know oh-so-well.

And so one required reading, which we’re positively dawdling over in class, is a blogpost which I’ve excerpted from below:

One could just as easily make fun of psychologists and sociologists for ignoring the rationality of much human decision-making, and price incentives in particular. Gary Becker made a splendid career out of that fact.  We could easily write parallel op-eds saying all of psychology is wrong because they omit the fact that sometimes people do in fact add two and two to get four. But “rationalists” respect logic and their reader’s intelligence too much to do that: Psychologists’ omissions and simplifications likewise do not invalidate their observations about other aspects of behavior.

https://johnhcochrane.blogspot.com/2015/05/homo-economicus-or-homo-paleas.html

Or as I tell my students, ask yourself why a field of study called “Rational Psychology” should not exist, now that you’re a student of “Behavioral Economics”. You see what I mean when I say it is a whole lot of fun!

But within this blogpost, there is one paragraph that I need your help with.

Behavioral marketing, for example, is a cornerstone of the business school curriculum. I presume Dick’s class “Managerial decision making” (syllabus sadly not available) covers a lot of how to use psychology to become more rational. Behavioral finance is excellent marketing for active investment strategies, that’s for sure.

https://johnhcochrane.blogspot.com/2015/05/homo-economicus-or-homo-paleas.html

I asked my students to check if they can find the syllabus John Cochrane is talking about in his blogpost. And on a whim, I decided to try and hunt it down myself. The blogpost was written in 2015, but it would seem that even now, the syllabus is not available online. Richard Thaler’s faculty page, linked to in the excerpt above, does contain the link to the course, but that page itself is just blank.

I dug around a little bit, but was only able to find a write-up about the course (but do read it, it is very, very interesting):

Memories of Thaler’s teaching style have one common thread for all of his former students: laughter. They can’t describe a Thaler class without a smile. “He meanders up to the podium,” said Linnea Gandhi, MBA ’14, who has known Thaler as a student, a teaching assistant, and now co-professor as adjunct assistant professor of behavioral science at Booth. “Then he tells stories about foibles and fumbles, not just in companies but in his own life.” In most students’ recollections, this is done at a slight lean, against the podium, against the transparency machine in the old days, against whatever’s handy.

https://www.chicagobooth.edu/magazine/class-behave-richard-thaler

And I cannot tell you how much I enjoyed reading about this, since I tried a variant of this method in my own class earlier this academic year (plus, he’s talking about ‘what are you optimizing for?’, a question I have fallen in love with):

He once ran into a student who was studying for the final exam in his class and said he was busy outlining the articles they had read, a thought that appalled Thaler. “I want them to have to think about it,” he said, “not just memorize what was said.” So he started using a new type of exam. He would ask students to submit potential exam questions and then would circulate about 75 of those questions, saying the exam would be composed of (slightly edited) versions of these. The rule in generating questions was that they could not have a simple “correct answer” but rather force the students to ponder the material they had learned and then apply it to some novel situation.
“It’s not the most precise way of measuring how much they’ve learned,” he said. “But it’s the best way I have found to maximize what they learn when studying for the exam.”

https://www.chicagobooth.edu/magazine/class-behave-richard-thaler

But anyways, here’s what I would appreciate help with, if possible. Do any of you have a copy of the syllabus in question? This search brings up quite a few syllabi, but not the specific one I’m looking for. Of course, to be clear, it may not be available online at all, and it certainly isn’t necessary that syllabi are always made available. In fact, if you ask me, a syllabus is a bit like a straitjacket – there is an argument to be made for not having one, and winging it in a semester. A blogpost for another day, this thought.

But for now, this bleg: a copy of the syllabus for Richard Thaler’s class on Managerial Decision Making, s’il vous plait.

Merci d’avance!

The End of the College Submission (Thank God)

This blog post is a riff on Seth’s post from the other day, titled “The End of the High School Essay“:

New York City schools are trying to ban GPT3 because it’s so good at writing superficial essays that it undermines the command structure of the essay as a sorting tool. An easy thing to assign (and a hard thing to grade) just became an easy task to hack.
High school essays had a huge range of problems, and banning the greatest essay device since Danny Dunn and his Homework Machine is not the answer. In fact, it’s a great opportunity to find a better way forward.
The first challenge of the essay was the asymmetrical difficulty in giving useful feedback. 30 essays, 5 minutes each, do the math. It doesn’t scale, and five minutes isn’t even close to enough time to honor the two hours you asked a student to put into the work.

https://seths.blog/2023/01/the-end-of-the-high-school-essay/

Exams in almost all of the colleges and universities I have taught at don’t mean a thing. The students know this, the faculty knows this, the examination department knows this, but we all keep up the charade that Meaningful Work Is Being Done through the conduct of examinations.

Newsflash: there is no meaningful work being done. It is a complete farce.

Some universities choose to not pay faculty members for correcting papers at the end of the semester. Let’s assume a college is paying a visiting faculty member two thousand rupees per hour to teach a class. They might slip in a line towards the end: this also includes examination duties. In English, this means that if you teach a thirty hour course, you will be paid sixty thousand rupees for those thirty hours. So far, so good. But “also includes examination duties” means that for a batch of (say) a hundred and twenty students, you are also expected to design question papers (a set of two, usually) and correct a hundred and twenty answer sheets.

Even if you assume that one is able to correct paper after paper without taking a break, with five minutes being the time taken per paper, that still means that at least ten hours worth of work. Which means, of course, that you are not being paid two thousand rupees per hour, but rather fifteen hundred. Accounting is a subject that may well be taught at universities – that doesn’t necessarily mean that it is practised at universities.

Some other universities offer to pay forty rupees per answer sheet corrected. Which is better than zero, admittedly, but we then run into the problem of incentives. If you’re paid two thousand rupees to teach, and forty rupees per paper to correct answer sheets, how many answer sheets should you correct in an hour to “make” the same wage? And if fifty answer sheets being corrected in an hour is clearly far too many, how do you expect this incentive to work? Or do we teach our students that incentives matter, but ignore this point ourselves?

Students know the farcical nature of examinations all too well. The pandemic took away that last remaining fig leaf of dignity that surrounds examinations, and the ostrich-in-the-sand approach that most universities have adopted post-pandemic is that of closed-book, no-internet-access examinations. Quite how this pen-and-paper examination is supposed to prepare students for what they will do in the real world is a question nobody wants to raise, let alone answer.

And so students quite reasonably ask for “the pattern of the paper”, or the “important questions” or the “important topics” before an examination. They are, in other words, seeking to minimize efforts in order to maximize marks scored in an examination. The tragedy lies in the fact that academia is supposed to be about maximizing learning. But on and on we go, in our mad headlong rush to maximize NAAC scores, difficult and uncomfortable questions about examinations be damned.

But all that these pen-and-paper examinations do is to train students to produce mediocre output that AI can already produce – and of a much better quality than these scribbled answers in answer sheets will ever produce. That’s not a knock against students; it is praise for how good AI has already gotten.

Think about it, for this is a point that bears repetition. Our examination system is geared towards training students to do a worse job than AI, by definition. And for this, we take money from students and their families, and we call it “an education”. Pah.

Now, I’m well aware of the fact that this is not applicable in all cases. There are some subjects/courses in the social sciences where these kind of examinations are entirely justified. And medical and engineering fields is a whole separate story. But I’m not arguing for an extreme solution – I’m saying that the pendulum has swung far too much over into Luddite territory when it comes to examinations and submissions. We need to wake up and smell the AI, and adjust accordingly.

But how? Well, the easy thing to do is to say that’s a difficult answer to give in a blogpost, but here’s Seth Godin again:

The answer is simple but difficult: Switch to the Sal Khan model. Lectures at home, classes are for homework.

When we’re on our own, our job is to watch the best lecture on the topic, on YouTube or at Khan Academy. And in the magic of the live classroom, we do our homework together.

In a school that’s privileged enough to have decent class sizes and devices in the classroom, challenge the students to actually discuss what they’ve read or learned. In real-time, teach them to not only create arguments but to get confident enough to refute them. Not only can the teacher ask a student questions, but groups of students can ask each other questions. Sure, they can use GPT or other tools to formulate where they begin, but the actual work is in figuring out something better than that.
At first, this is harder work for the teacher, but in fact, it’s what teachers actually signed up to do when they become teachers.

This is far less cohesive and controllable than the industrial model of straight rows and boring lectures. It will be a difficult transition indeed. But it’s simple to think about: If we want to train people to take initiative, to question the arguments of others, to do the reading and to create, perhaps the best way to do that is to have them do that.

We’ll never again need to hire someone to write a pretty good press release, a pretty good medical report or a pretty good investor deck. Those are instant, free and the base level of mediocre. The opportunity going forward remains the same: Bringing insight and guts to interesting problems.

https://seths.blog/2023/01/the-end-of-the-high-school-essay/

Kill our current mode of examinations, and help build a world in which we have passionate teachers who help students create. Not a world in which we minimize soul, and maximize those stupid, accursed “marks”.

But on and on we go. Pah.

Do Less Of

Here is a list of things I hope to do less of in 2023:

  1. Exams (setting papers, correcting papers). Exams, given the way they are conducted almost everywhere I teach in India, are stupid. They fill hours of my time, but neither I, nor the students, benefit from them in any way. I am more than willing to die on this hill. Exams are stupid.
  2. Eat lesser. But conversely and not at all paradoxiacally, make every meal count. If the option is between eating a badly made meal and not eating, I hope I have the wisdom to pick the latter every time. It will have at least two advantages – I get to fast more than I did this year, and when I do eat, it is likely to be a much more pleasant experience.
  3. Spend less time reading stuff on the phone. My attention span suffers for it, and I need to get better at not taking my phone out of my pocket every chance I get.
  4. Reading while I am eating. Related to the second point, of course – I aim to make every meal I eat a celebration of food – but this is also related to the third point. But in general, try to get out of the habit of using the phone as a way to escape the world around me.
  5. Mindless YouTubing. That’s not the same as spending less time on YouTube – I’m convinced that I need to spend more time on the platform. But YouTube Shorts is the enemy, and so is YouTube’s recommendation engine. It doesn’t optimize for what is best for me in the long run, and I think Odysseus may well have been on to something.

Make Examinations Relevant Again

Alice Evans (and if you are unfamiliar with her work, here’s a great way to begin learning more about it) recently tweeted about a topic that is close to my heart:

And one of the replies was fascinating:

I’ve asked students to create podcasts in the past for assignments, but not yet for final or semester end examinations, because I am constrained by the rules of whichever university I’m teaching in. There are some that allow for experimentation and off-the-beaten-path formats, but the vast majority are still in “Answer the following” mode.

But ever since I came across that tweet, I’ve been thinking about how we could make examinations in this country better, more relevant, and design them in such a way that we test skills that are applicable to the world we live in today, rather than the world of a 100 years ago.

To me, the ideal examination would include the following:

  • The ability to do fast-paced research on a collaborative basis
  • The ability to work as a team to be able to come up with output on the basis of this research
  • The ability to write (cogently and concisely) about how you as an individual think about the work that your team came up with

What might such an examination look like? Well, it could take many forms, but here’s one particular form that I have been thinking about.

Imagine an examination for a subject like, say, macroeconomics. Here’s a question I would love to ask students to think about for such an examination today. “Do you and your team find yourself on Team Transitory or Team Persistent when it comes to inflation today? The answer, in whatever format, should make sense to a person almost entirely unacquainted with economics.”

This would be a three hour long examination. Say the exam is for a cohort of 120 students. I’d split the class up into 10 groups of 12 each, and ask each group to spend one hour thinking about this question, and doing the research necessary to come up with an answer. They can discuss the question, split the work up (refer to textbooks, refer to material online, watch YouTube videos, discuss with each other, appoint a leader – whatever it is that they need to do) and come up with an outline of what their answer is.

The next hour would be coming up with the answer itself: write a blogpost about it, or record audio, or record video. The format is up to them, as is the length. The only requirement is that the output must answer the question, and must include reasons for their choice. Whether the background information that is required to make sense is to be given (or referenced, or skipped altogether) is entirely up to the students.

And the final hour must be spent on a short write-up where each individual student submits their view about their team’s submission. Given that the second hour’s output was collaborative, does the student as an individual agree with the work done? Why? Or why not? What would the student have liked to have done differently? This part must be written, for the ability to write well is (to me) non-negotiable.

To me, this examination will encompass research (which can only be done in an hour if the students are familiar enough with the subject at hand, so they need to have done their homework), collaboration and the ability to think critically about the work that they were a part of. Grading could be split equally on a fifty-fifty basis: half for the work done collaboratively, and half for the individual essay submission.

Sure, there would be some problems. Students might object to the groups that have been formed or students might end up quarreling so much in the first two hours that they’re not left with much time. Or something else altogether, which is impossible to foresee right now.

But I would argue that such examinations are more reflective of the work that the students will actually do in the world outside. More reflective than “Answer the following” type questions, that is.

The point isn’t to defend this particular format. The point is to ask if the current format needs to change (yes!) and if so how (this being only one suggestion).

Right now, examinations provide a 19th century solution to very real 21st century problems, and their irrelevance becomes ever more glaring by the day.

We need to talk about examinations, and we aren’t.

But What Will We Do?

All that is well and good: a high quality, low scale, not very cheap university that gives away it’s plans and implementation details for free. But what would students and faculty in such a university, well, do?

There are two ways to answer this question.

First, work out what they do in most universities today, reach a conclusion about whether what they do is desirable, and if not, work out what needs to change and in what order.

Second, start with an idealized worldview of what they should do, while ignoring all constraints, and ask if that idealized worldview is realizable. If the answer is no, figure out what’s stopping you, and therefore work out what needs to change and in what order.

I prefer the first approach((why is a blogpost in its own right)), and that’s the approach I’ll be using in this essay.

So what do students in a university do today? They learn, they show that they’ve learnt, and they form networks. College is a bundle.

Let’s begin with the one in the middle: showing that they’ve learnt. I’d want to chip away at that first.

How do students today show that they’ve learnt? They write examinations. These are supposed to be a proxy to show how much you’ve learnt, and how much you’re able to apply of whatever you’ve learnt. Is my understanding of why examinations exist correct? I’m genuinely asking.

And the reason I am asking is because I am extremely sceptical of the ability of examinations to do either.

That leaves us with two potential answers: change the examination system for the better. Or replace it with something else that has the potential to be better, and a guarantee of not being worse. At the very least, experimentation is called for.

My personal preference will be for a student to do, not for a student to show that she has learnt. That is, examinations need to be replaced with projects. These projects can’t be submissions as you and I understand them today. Not half-assed stuff that a team “works” on – let’s be honest, that’s code for nobody works on it. This is a project that sits on its own individual website, and that forever. That project is the student’s CV. It contains everything about the journey that led up to the creation and shipping of that project.

I’ll use an example from the field in which I teach, which is economics. What I am about to say should also apply to most of the other subjects in the humanities, and perhaps less so in, say, STEM fields.

Most students of economics acquire a degree to usually do one of three things: teach, research or work in a corporate job.

So your only “examination” comes at the end of your third year degree, where you ship a project based upon one of these three areas.

  • If you are learning economics in order to be a teacher, you have to teach a class, and document that entire class on your website. And when I say class, I mean at least a twenty hour course. Subject? Your choice. Students? You have to recruit them. Will this class be for free? No, you have to convince your students to pay a fee that is not trifling. How many students? At least ten, not more than thirty. But your degree is awarded for having taught this class, and is based on the feedback from that class.
    Designing the syllabus for that class, coming up with the reading material, arranging for guest speakers, hiring your TA’s from among your juniors, coming up with assignments, evaluating these assignments, handouts – the whole shebang. Do the work, teach your students, and then say that you have graduated from the course.
    And the entire three year journey leads up to this moment. If what you want to do is teach, then we should have taught you how to teach – and what skills you need to pick up in order to be able to teach students who are willing to pay you for your expertise.
  • If you are learning economics in order to do research, then you have to work on and publish a report on a topic of your choice. Come up with a topic in your second year, refine it, apply for a grant for it, prepare a plan for how you will work towards it, hire out your team, design your questionnaire, collect your data, clean the data, do the analysis, reach your conclusion and submit the report and the presentation to the funding agency, and on your website.
    Again, the entire three year journey leads up to this moment. If what you want to do is research, then we should have taught you how to do research – and what skills you need to pick up in order to be able to do research for an agency that is willing to pay you for your expertise.
  • If you are learning economics in order to work in a firm or start a business of your own, then, well, you have to do these things to get the degree. You have to convince a firm to hire you for an entire semester, or you have to spend an entire semester building out a team, pitching for funding, and get a product to market in order to be awarded a degree.

Long story short, you are evaluated for what you have done, not for what you have submitted for internal review and assessment.

My Almost Ideal University would have no examinations, but only a specific end-goal. Do the work, and if you’ve done it well enough, you’re awarded the degree. There’s no first place, no last place, no grades, no marks, nothing. If your work is good enough, we say that you are good enough to go out and start doing more of what you just finished doing.

Will this system have problems in terms of implementation. It’s a guarantee. Can I pick flaws in the design that I have put up? A dozen.

But does it have a fighting chance of being, at least along some dimensions, better than the status quo? Even if you happen to disagree, I think it to be a question worthy of further discussion. So especially if you disagree, please, do tell me why! 🙂

So that’s what the students will do. What will “faculty” do in my Almost Ideal University? We’ll talk about it tomorrow!

About Innovative Question Papers

The typical exams, at least in Indian universities, tend to be really dull, drab affairs. I cannot speak about how good (or bad) they are in other countries, but partly by design, partly due to inertia, they follow the same old template in most Indian universities.

You’ll get your fair share of “State briefly”, a sprinkling of “Explain why”, and the very occasional “In your opinion”. But for the most part, it needs rote memorization to respond and “do well” in these examinations. Take home essays, reflective essays and anything that is even remotely innovative are not welcome.

But in a world where such a thing was possible, what might question papers look like?

Tyler [Cowen] once walked into class the day of the final exam and he said. “Here is the exam. Write your own questions. Write your own answers. Harder questions and better answers get more points.” Then he walked out. The funniest thing was when a student came in late and I had to explain to him what the exam was and he didn’t believe me!

https://chrisblattman.com/2012/08/15/best-exam-question-ever/
https://chrisblattman.com/files/2013/01/1-Introduction.pdf

I can’t find it online right now, but I seem to remember that some of Prof. Blattman’s papers included the additional condition that you needed to submit the cheat sheet, and it too would be evaluated for what you chose to put in it (and implicitly, what you chose to leave out).

I had once asked a very small development economics class to teach some of their peers – those who hadn’t taken Dev Eco – the Solow Model. The non-math version, of course. And then those who had been taught had to take a small multiple choice question test. The marks they scored would become the grades for the “teachers”. Stuff like this will never work for a larger class, will never work for a semester end exam, but it still was a lot of fun to do!

The point is that it is all too possible to have examinations be fun, and be about learning. But so long as examinations are about signaling how much you’ve scored, rather than allowing you to reflect on how much you’ve learnt, they’ll have to be standardized, which means they’ll have to continue to be boring.

If you’d like them to be fun, you need to try and live in a world where examinations (and therefore colleges) aren’t about signaling.

But that is a tall order, no?

Proposed Examination Reforms

I’m not holding my breath, but this article has raised my hopes just a little bit:

Colleges and universities may soon adopt continuous comprehensive evaluation, a method that shifts focus from only annual or semester-level summative assessment system.
The suggestion has come from higher education regulator University Grants Commission (UGC) amid the increasing dependence on technology for education delivery in the current pandemic environment. Assessment at several intervals during and after achievement of learning outcomes specified for every module is needed as blended learning is gaining ground, UGC said in a draft proposal shared with higher educational institutions.

https://www.livemint.com/education/news/ugc-bats-for-reforms-in-exams-with-a-focus-on-continuous-evaluation-11621798197326.html

I’ve said it before, and I’ll say it again: everybody associated with academia in India knows how broken, pointless and screwed-up examinations are, but nobody wants to do anything about it. And the most often quoted reason is R Madhavan in 3 Idiots going “Abba nahi manenge”.

Abba being the UGC.

But now, hallelujah, the UGC is talking about open book examinations and on-demand examinations. This was a “tears in my eyes moment“:

Open book exams is the “right way to move away from the conventional approach of exam where remembering and reproducing is prime”, UGC said. “In real functioning beyond formal education, life is all about open book examination. Hence, in higher education, we must prepare students for work life by making them acquainted with open book examinations. It will also facilitate better understanding and application of knowledge,” UGC said, citing an internal committee report.

https://www.livemint.com/education/news/ugc-bats-for-reforms-in-exams-with-a-focus-on-continuous-evaluation-11621798197326.html

There is still a world of pain that awaits those of us who are in academia. The inertia associated with the old system will take years((And if I am to be cynical, which is almost always the case, I’ll say decades)) to overcome, and it will be a long, unpleasant journey.

More, we will run up against capacity constraints, because shifting away from the “State in brief” questions to having students think critically will require the changing of multiple mindsets, along with intensive training of faculty in all universities.

And even if some universities were to adopt this whole-heartedly, one unintended consequence will be the exacerbation of already ridiculously high inequality. The inequality I speak of is in terms of access to quality higher education, of course. Better colleges and universities will get better still, and while that is desirable for the students who are lucky enough to get into them, it doesn’t bode well for equitable educational outcomes across the country.

But even so, the very fact that this is even being discussed in the first place is a welcome move.

The one thing that gives me hope is something that is discussed almost as an after-thought in the article: e-portfolios.

An electronic porfolio (e-portfolio) is a purposeful collection of sample student work, demonstrations, and artifacts that showcase student’s learning progression, achievement, and evidence of what students can do. The collection can include essays and papers (text-based), blog, multimedia (recordings of demonstrations, interviews, presentations, etc.), graphic.

https://teaching.berkeley.edu/resources/assessment-and-evaluation/design-assessment/e-portfolio

This blog, for example, is my “e-portfolio”. I pay around ten to twelve thousand rupees every year to maintain this blog, but one can of course start a blog for free. Or a YouTube channel, or an Instagram page or absolutely anything else you like.

In an ideal world, e-portfolios (and could we come up with e better name for it, please?) are solely the responsibility of the student. They can be in any language. They can be nurtured over time, for years together. Cultivating your e-portfolio needn’t cost money, in other words, and popularizing your e-portfolio is a life-skill worth developing in its own right.

Most importantly, developing one requires just a smartphone. Yes, this is still a challenge for large parts of our country, but I would argue that a learning system that revolves around the development of an e-portfolio is more efficient, cheaper and easier than even a perfectly reformed examination system.

Bottomline: marks, examinations and degrees are overrated. Doing the work, and sharing your work in the public domain is underrated.

Here’s a blogpost from last year about conducting examinations during these crazy times, and here are all the posts I have written about higher education on EFE.

On Conducting Examinations, Especially in 2020

This is less a blogpost, and more of a rant. Consider yourself warned! 🙂

This past Saturday, I cam across a most excellent Twitter thread:

https://twitter.com/CT_Bergstrom/status/1322369355930165248

The author, Carl Bergstrom (Wikipedia article here, University profile here) makes a detailed, reasoned argument against online proctoring of examinations, especially in 2020. In my blogpost, I’m going to riff on this Twitter thread, and some related points, and build an argument against the way we conduct examinations in Indian Universities.

To begin, watch this Sugata Mitra TED talk about education:

Just the first five minutes or so is enough for the argument I will be making today, but you really should watch the full thing. There was, as it turns out, a case for rote memorization at one point of time. But today, as Mitra says in the video, it is the computers that are the clerks. They do (and to be fair to them, they do it much better than we ever could) the job of remembering everything, so that we don’t have to.

To every single professor reading this blogpost: when was the last time you yourself spent a day working without looking up something on the internet? When was the last time you researched something, wrote something, without using your computer, or the internet on some device?

Then why do we insist on examining our students for their ability to do so, when we ourselves don’t do it? And for those of our students who are not going to get into academia, they wouldn’t last in their organizations for even an hour if they tried to work without using computers and the internet.

They would (should!) be fired for being Luddites!

And yet, to land up in that firm, they must spend the last week of their lives as a college student cooped up for three hours in a classroom without a computer, without the internet, and use pen and paper to write out the important features of xyz in abc lines.

I can’t possibly be the only one that spots the incongruity, surely?

Here, from the Twitter thread I spoke about earlier, is a picture of Bloom’s taxonomy:

Image

Read the Wikipedia article about Bloom’s taxonomy, or look carefully at the picture above. At best – and I think I am being charitable here – our question papers in higher education in India reach the evaluate stage, but certainly not create. And even that is a stretch.

Moreover, even if we somehow agree that we do indeed reach the evaluate stage, we are effectively asking students to evaluate based on their memory alone. Why?

One, won’t students write a better evaluation of whatever theory we are asking them to write about if we give them the ability to research while writing? Second – and I know this is repetitive, but still – are they ever going to write an evaluation without having access to to the internet?

In plain simple English: We train students for 25 years to get awesome at memorizing stuff, and then expect them to do well in a world which doesn’t value this skill at all.

(To be clear, some things you should remember, of course. Think of it as a spectrum – and I am not suggesting that we move to the end of the spectrum where no memorization is required. I am suggesting, however, that we are at the end of the spectrum where only memorization is required. Close enough, at any rate).

Coming specifically to this year, the year of online examinations, here’s a tweet that was quoted in Bergstrom’s thread:

There really isn’t much to say, is there? All universities the world over have sent out variations of this nightmare this year, and in some cases, repeatedly. It’s the whole null hypothesis argument all over again – we assume all students to be guilty until proven otherwise. That is, we assume everybody will cheat, and therefore force everybody to comply with ridiculously onerous rules – so as to prevent the few who might actually cheat.

And cheating, of course, being looking up stuff on the internet. The argument itself is pointless, as I have explained above, and we go to eye-popping lengths to enforce the logical outcome of this pointless argument.

Prof. Bergstrom makes the same points himself in the Twitter thread, of course:

https://twitter.com/CT_Bergstrom/status/1322372971730644992
https://twitter.com/CT_Bergstrom/status/1322375318108213248
https://twitter.com/CT_Bergstrom/status/1322376239164743680

This year, especially, is a good opportunity to turn what is otherwise a disaster of a situation into meaningful reform of the way we conduct examinations.

Students, parents – indeed society at large – will spot the incongruity of learning online, but being examined offline. If we, in higher education in India do not spot this incongruity and work towards changing it – well then, we will have failed.

And finally, the last tweet in the thread is something we would all do well to remember:

https://twitter.com/CT_Bergstrom/status/1322378737464172547