Here is a list of things I hope to do less of in 2023:
Exams (setting papers, correcting papers). Exams, given the way they are conducted almost everywhere I teach in India, are stupid. They fill hours of my time, but neither I, nor the students, benefit from them in any way. I am more than willing to die on this hill. Exams are stupid.
Eat lesser. But conversely and not at all paradoxiacally, make every meal count. If the option is between eating a badly made meal and not eating, I hope I have the wisdom to pick the latter every time. It will have at least two advantages – I get to fast more than I did this year, and when I do eat, it is likely to be a much more pleasant experience.
Spend less time reading stuff on the phone. My attention span suffers for it, and I need to get better at not taking my phone out of my pocket every chance I get.
Reading while I am eating. Related to the second point, of course – I aim to make every meal I eat a celebration of food – but this is also related to the third point. But in general, try to get out of the habit of using the phone as a way to escape the world around me.
Mindless YouTubing. That’s not the same as spending less time on YouTube – I’m convinced that I need to spend more time on the platform. But YouTube Shorts is the enemy, and so is YouTube’s recommendation engine. It doesn’t optimize for what is best for me in the long run, and I think Odysseus may well have been on to something.
I have helped run an AI-based entrepreneurship program for years, written papers on the econ of AI, and follow the field quite closely. Nonetheless, I am *shocked* by how good OpenAI's new chat (https://t.co/6PbtqZUziI) is. E.g., you can no longer give take-home exams/homework.
Instead of a final paper or exam, I have students prepare a 10-minute podcast, working in pairs. My daughter, who produces a podcasts for NPR, gives a guest lecture on techniques. Results are great and they like it.
I’ve asked students to create podcasts in the past for assignments, but not yet for final or semester end examinations, because I am constrained by the rules of whichever university I’m teaching in. There are some that allow for experimentation and off-the-beaten-path formats, but the vast majority are still in “Answer the following” mode.
But ever since I came across that tweet, I’ve been thinking about how we could make examinations in this country better, more relevant, and design them in such a way that we test skills that are applicable to the world we live in today, rather than the world of a 100 years ago.
To me, the ideal examination would include the following:
The ability to do fast-paced research on a collaborative basis
The ability to work as a team to be able to come up with output on the basis of this research
The ability to write (cogently and concisely) about how you as an individual think about the work that your team came up with
What might such an examination look like? Well, it could take many forms, but here’s one particular form that I have been thinking about.
Imagine an examination for a subject like, say, macroeconomics. Here’s a question I would love to ask students to think about for such an examination today. “Do you and your team find yourself on Team Transitory or Team Persistent when it comes to inflation today? The answer, in whatever format, should make sense to a person almost entirely unacquainted with economics.”
This would be a three hour long examination. Say the exam is for a cohort of 120 students. I’d split the class up into 10 groups of 12 each, and ask each group to spend one hour thinking about this question, and doing the research necessary to come up with an answer. They can discuss the question, split the work up (refer to textbooks, refer to material online, watch YouTube videos, discuss with each other, appoint a leader – whatever it is that they need to do) and come up with an outline of what their answer is.
The next hour would be coming up with the answer itself: write a blogpost about it, or record audio, or record video. The format is up to them, as is the length. The only requirement is that the output must answer the question, and must include reasons for their choice. Whether the background information that is required to make sense is to be given (or referenced, or skipped altogether) is entirely up to the students.
And the final hour must be spent on a short write-up where each individual student submits their view about their team’s submission. Given that the second hour’s output was collaborative, does the student as an individual agree with the work done? Why? Or why not? What would the student have liked to have done differently? This part must be written, for the ability to write well is (to me) non-negotiable.
To me, this examination will encompass research (which can only be done in an hour if the students are familiar enough with the subject at hand, so they need to have done their homework), collaboration and the ability to think critically about the work that they were a part of. Grading could be split equally on a fifty-fifty basis: half for the work done collaboratively, and half for the individual essay submission.
Sure, there would be some problems. Students might object to the groups that have been formed or students might end up quarreling so much in the first two hours that they’re not left with much time. Or something else altogether, which is impossible to foresee right now.
But I would argue that such examinations are more reflective of the work that the students will actually do in the world outside. More reflective than “Answer the following” type questions, that is.
The point isn’t to defend this particular format. The point is to ask if the current format needs to change (yes!) and if so how (this being only one suggestion).
Right now, examinations provide a 19th century solution to very real 21st century problems, and their irrelevance becomes ever more glaring by the day.
We need to talk about examinations, and we aren’t.
This is a continuation of a series. The first post, this Monday, asked how we might transition from online to offline education when (if?) the pandemic ends. The second post was about me trying to figure out in which ways offline classes are better. This post is about me trying to figure out ways in which offline examinations are better.
Offline examinations, in the context of this post, are defined as examinations in which students sit in a classroom for three hours, and write detailed answers using pen and paper, without having access to their textbooks or to internet enabled devices.
They aren’t better.
That’s it.
I cannot tell you how strongly I feel about this. Note that this post is about higher education, not about school level exams. But that being said, the idea that an offline examination replicates real life conditions is patently ridiculous.
When was the last time, in the course of your normal workday, that you sat in a room in which you couldn’t access the help of your colleagues or the internet, with only pen and paper, and did work? And even if you were to say to me that such a thing has happened, did that work involve regurgitating what you already know? Or was that work about generating new ideas without being distracted by the internet?
Offline examinations are not about generating new ideas. They aren’t about testing how well you would do in a realistic work setting. I honestly do not know what they are about, and I cannot for the life of me understand why they existed up until covid-19 came knocking.
Offline examinations need to go, and I would love to learn why I am wrong about this. Please help me understand.
All that is well and good: a high quality, low scale, not very cheap university that gives away it’s plans and implementation details for free. But what would students and faculty in such a university, well, do?
There are two ways to answer this question.
First, work out what they do in most universities today, reach a conclusion about whether what they do is desirable, and if not, work out what needs to change and in what order.
Second, start with an idealized worldview of what they should do, while ignoring all constraints, and ask if that idealized worldview is realizable. If the answer is no, figure out what’s stopping you, and therefore work out what needs to change and in what order.
I prefer the first approach((why is a blogpost in its own right)), and that’s the approach I’ll be using in this essay.
So what do students in a university do today? They learn, they show that they’ve learnt, and they form networks. College is a bundle.
Let’s begin with the one in the middle: showing that they’ve learnt. I’d want to chip away at that first.
How do students today show that they’ve learnt? They write examinations. These are supposed to be a proxy to show how much you’ve learnt, and how much you’re able to apply of whatever you’ve learnt. Is my understanding of why examinations exist correct? I’m genuinely asking.
That leaves us with two potential answers: change the examination system for the better. Or replace it with something else that has the potential to be better, and a guarantee of not being worse. At the very least, experimentation is called for.
My personal preference will be for a student to do, not for a student to show that she has learnt. That is, examinations need to be replaced with projects. These projects can’t be submissions as you and I understand them today. Not half-assed stuff that a team “works” on – let’s be honest, that’s code for nobody works on it. This is a project that sits on its own individual website, and that forever. That project is the student’s CV. It contains everything about the journey that led up to the creation and shipping of that project.
I’ll use an example from the field in which I teach, which is economics. What I am about to say should also apply to most of the other subjects in the humanities, and perhaps less so in, say, STEM fields.
Most students of economics acquire a degree to usually do one of three things: teach, research or work in a corporate job.
So your only “examination” comes at the end of your third year degree, where you ship a project based upon one of these three areas.
If you are learning economics in order to be a teacher, you have to teach a class, and document that entire class on your website. And when I say class, I mean at least a twenty hour course. Subject? Your choice. Students? You have to recruit them. Will this class be for free? No, you have to convince your students to pay a fee that is not trifling. How many students? At least ten, not more than thirty. But your degree is awarded for having taught this class, and is based on the feedback from that class. Designing the syllabus for that class, coming up with the reading material, arranging for guest speakers, hiring your TA’s from among your juniors, coming up with assignments, evaluating these assignments, handouts – the whole shebang. Do the work, teach your students, and then say that you have graduated from the course. And the entire three year journey leads up to this moment. If what you want to do is teach, then we should have taught you how to teach – and what skills you need to pick up in order to be able to teach students who are willing to pay you for your expertise.
If you are learning economics in order to do research, then you have to work on and publish a report on a topic of your choice. Come up with a topic in your second year, refine it, apply for a grant for it, prepare a plan for how you will work towards it, hire out your team, design your questionnaire, collect your data, clean the data, do the analysis, reach your conclusion and submit the report and the presentation to the funding agency, and on your website. Again, the entire three year journey leads up to this moment. If what you want to do is research, then we should have taught you how to do research – and what skills you need to pick up in order to be able to do research for an agency that is willing to pay you for your expertise.
If you are learning economics in order to work in a firm or start a business of your own, then, well, you have to do these things to get the degree. You have to convince a firm to hire you for an entire semester, or you have to spend an entire semester building out a team, pitching for funding, and get a product to market in order to be awarded a degree.
Long story short, you are evaluated for what you have done, not for what you have submitted for internal review and assessment.
My Almost Ideal University would have no examinations, but only a specific end-goal. Do the work, and if you’ve done it well enough, you’re awarded the degree. There’s no first place, no last place, no grades, no marks, nothing. If your work is good enough, we say that you are good enough to go out and start doing more of what you just finished doing.
Will this system have problems in terms of implementation. It’s a guarantee. Can I pick flaws in the design that I have put up? A dozen.
But does it have a fighting chance of being, at least along some dimensions, better than the status quo? Even if you happen to disagree, I think it to be a question worthy of further discussion. So especially if you disagree, please, do tell me why! 🙂
So that’s what the students will do. What will “faculty” do in my Almost Ideal University? We’ll talk about it tomorrow!
I’m not holding my breath, but this article has raised my hopes just a little bit:
Colleges and universities may soon adopt continuous comprehensive evaluation, a method that shifts focus from only annual or semester-level summative assessment system. The suggestion has come from higher education regulator University Grants Commission (UGC) amid the increasing dependence on technology for education delivery in the current pandemic environment. Assessment at several intervals during and after achievement of learning outcomes specified for every module is needed as blended learning is gaining ground, UGC said in a draft proposal shared with higher educational institutions.
I’ve said it before, and I’ll say it again: everybody associated with academia in India knows how broken, pointless and screwed-up examinations are, but nobody wants to do anything about it. And the most often quoted reason is R Madhavan in 3 Idiots going “Abba nahi manenge”.
Abba being the UGC.
But now, hallelujah, the UGC is talking about open book examinations and on-demand examinations. This was a “tears in my eyes moment“:
Open book exams is the “right way to move away from the conventional approach of exam where remembering and reproducing is prime”, UGC said. “In real functioning beyond formal education, life is all about open book examination. Hence, in higher education, we must prepare students for work life by making them acquainted with open book examinations. It will also facilitate better understanding and application of knowledge,” UGC said, citing an internal committee report.
There is still a world of pain that awaits those of us who are in academia. The inertia associated with the old system will take years((And if I am to be cynical, which is almost always the case, I’ll say decades)) to overcome, and it will be a long, unpleasant journey.
More, we will run up against capacity constraints, because shifting away from the “State in brief” questions to having students think critically will require the changing of multiple mindsets, along with intensive training of faculty in all universities.
And even if some universities were to adopt this whole-heartedly, one unintended consequence will be the exacerbation of already ridiculously high inequality. The inequality I speak of is in terms of access to quality higher education, of course. Better colleges and universities will get better still, and while that is desirable for the students who are lucky enough to get into them, it doesn’t bode well for equitable educational outcomes across the country.
But even so, the very fact that this is even being discussed in the first place is a welcome move.
The one thing that gives me hope is something that is discussed almost as an after-thought in the article: e-portfolios.
An electronic porfolio (e-portfolio) is a purposeful collection of sample student work, demonstrations, and artifacts that showcase student’s learning progression, achievement, and evidence of what students can do. The collection can include essays and papers (text-based), blog, multimedia (recordings of demonstrations, interviews, presentations, etc.), graphic.
This blog, for example, is my “e-portfolio”. I pay around ten to twelve thousand rupees every year to maintain this blog, but one can of course start a blog for free. Or a YouTube channel, or an Instagram page or absolutely anything else you like.
In an ideal world, e-portfolios (and could we come up with e better name for it, please?) are solely the responsibility of the student. They can be in any language. They can be nurtured over time, for years together. Cultivating your e-portfolio needn’t cost money, in other words, and popularizing your e-portfolio is a life-skill worth developing in its own right.
Most importantly, developing one requires just a smartphone. Yes, this is still a challenge for large parts of our country, but I would argue that a learning system that revolves around the development of an e-portfolio is more efficient, cheaper and easier than even a perfectly reformed examination system.
Bottomline: marks, examinations and degrees are overrated. Doing the work, and sharing your work in the public domain is underrated.
The author, Carl Bergstrom (Wikipedia article here, University profile here) makes a detailed, reasoned argument against online proctoring of examinations, especially in 2020. In my blogpost, I’m going to riff on this Twitter thread, and some related points, and build an argument against the way we conduct examinations in Indian Universities.
To begin, watch this Sugata Mitra TED talk about education:
Just the first five minutes or so is enough for the argument I will be making today, but you really should watch the full thing. There was, as it turns out, a case for rote memorization at one point of time. But today, as Mitra says in the video, it is the computers that are the clerks. They do (and to be fair to them, they do it much better than we ever could) the job of remembering everything, so that we don’t have to.
To every single professor reading this blogpost: when was the last time you yourself spent a day working without looking up something on the internet? When was the last time you researched something, wrote something, without using your computer, or the internet on some device?
Then why do we insist on examining our students for their ability to do so, when we ourselves don’t do it? And for those of our students who are not going to get into academia, they wouldn’t last in their organizations for even an hour if they tried to work without using computers and the internet.
They would (should!) be fired for being Luddites!
And yet, to land up in that firm, they must spend the last week of their lives as a college student cooped up for three hours in a classroom without a computer, without the internet, and use pen and paper to write out the important features of xyz in abc lines.
I can’t possibly be the only one that spots the incongruity, surely?
Read the Wikipedia article about Bloom’s taxonomy, or look carefully at the picture above. At best – and I think I am being charitable here – our question papers in higher education in India reach the evaluate stage, but certainly not create. And even that is a stretch.
Moreover, even if we somehow agree that we do indeed reach the evaluate stage, we are effectively asking students to evaluate based on their memory alone. Why?
One, won’t students write a better evaluation of whatever theory we are asking them to write about if we give them the ability to research while writing? Second – and I know this is repetitive, but still – are they ever going to write an evaluation without having access to to the internet?
In plain simple English: We train students for 25 years to get awesome at memorizing stuff, and then expect them to do well in a world which doesn’t value this skill at all.
(To be clear, some things you should remember, of course. Think of it as a spectrum – and I am not suggesting that we move to the end of the spectrum where no memorization is required. I am suggesting, however, that we are at the end of the spectrum where only memorization is required. Close enough, at any rate).
Coming specifically to this year, the year of online examinations, here’s a tweet that was quoted in Bergstrom’s thread:
Here are the requirements for my math midterm next week. You tell me if these instructions are easy to follow and don't make you nervous. pic.twitter.com/WnW0YSXmhr
There really isn’t much to say, is there? All universities the world over have sent out variations of this nightmare this year, and in some cases, repeatedly. It’s the whole null hypothesis argument all over again – we assume all students to be guilty until proven otherwise. That is, we assume everybody will cheat, and therefore force everybody to comply with ridiculously onerous rules – so as to prevent the few who might actually cheat.
And cheating, of course, being looking up stuff on the internet. The argument itself is pointless, as I have explained above, and we go to eye-popping lengths to enforce the logical outcome of this pointless argument.
Prof. Bergstrom makes the same points himself in the Twitter thread, of course:
This year, especially, is a good opportunity to turn what is otherwise a disaster of a situation into meaningful reform of the way we conduct examinations.
Students, parents – indeed society at large – will spot the incongruity of learning online, but being examined offline. If we, in higher education in India do not spot this incongruity and work towards changing it – well then, we will have failed.
And finally, the last tweet in the thread is something we would all do well to remember:
EconForEverybody 