Are offline exams better? No.

This is a continuation of a series. The first post, this Monday, asked how we might transition from online to offline education when (if?) the pandemic ends. The second post was about me trying to figure out in which ways offline classes are better. This post is about me trying to figure out ways in which offline examinations are better.

Offline examinations, in the context of this post, are defined as examinations in which students sit in a classroom for three hours, and write detailed answers using pen and paper, without having access to their textbooks or to internet enabled devices.

They aren’t better.

That’s it.

I cannot tell you how strongly I feel about this. Note that this post is about higher education, not about school level exams. But that being said, the idea that an offline examination replicates real life conditions is patently ridiculous.

When was the last time, in the course of your normal workday, that you sat in a room in which you couldn’t access the help of your colleagues or the internet, with only pen and paper, and did work? And even if you were to say to me that such a thing has happened, did that work involve regurgitating what you already know? Or was that work about generating new ideas without being distracted by the internet?

Offline examinations are not about generating new ideas. They aren’t about testing how well you would do in a realistic work setting. I honestly do not know what they are about, and I cannot for the life of me understand why they existed up until covid-19 came knocking.

Offline examinations need to go, and I would love to learn why I am wrong about this. Please help me understand.

The Economist on Year Three of the Pandemic

(Note: this was written and scheduled for posting before the world found out about Omicron. I have not changed a single word, except for the two sentences in these brackets)

‘Tis that time of the year, and we will soon be inundated with reflections on the year gone by, and the year to come. The Economist has come up with its list, and today, we will be focusing on one from among this series: What To Expect in Year Three of the Pandemic.

  1. The key takeaway is that the world as a whole will be better off because of the vaccines that become widely available in 2021, but…
  2. Vaccine inequity, already unfortunately visible, will become starker still. And this will have obvious ramifications on health (that much is obvious), but also on economic outcomes.
    “A disparity of outcomes between rich and poor countries will emerge. The Gates Foundation, one of the world’s largest charities, predicts that average incomes will return to their pre-pandemic levels in 90% of advanced economies, compared with only a third of low- and middle-income economies.”
  3. Distribution difficulties and vaccine hesitancy will also play (unfortunate) roles in the continuing saga, and a glut (imagine!) is not impossible to imagine in late 2022
  4. This is a chart well worth staring at. I encourage you to stare at it:

5. Vaccines will become better, more broad based, and supply chains will ease out in part because of technological advancements, such as freeze-dried mRNA vaccines.

6. But the larger point that I personally take away from the article is this: it’s going to be better, the article says than both 2020 and 2021, but it won’t be over in 2022. Variants will emerge, hesitancy will remain, and inequity will persist.
There will be, in other words, progress, but not as much as one would have liked, not as fast as one would have liked, and with complications that are bound to emerge, but impossible to currently specify.
Better, in short, but not by much, and with real risks to boost.

7. But all that being said, given the year that went by, I suppose we should take what we get.

India’s Demographics in One Tweet

Well, ok, not India’s demographics in one tweet, maybe. But it is such telling and thought-provoking trivia, this.

If you’re looking for a frame of reference, Belgium’s total population is 12 million. We will add 12 million 18 year olds alone.

By the way, please don’t misconstrue my stance on the issue: I’m very much on Team Simon.

Noah Smith Interviews Marc Andreessen, Part II

We pick up from where we left off yesterday:

My partner Alex Rampell says that competition between an incumbent and a software-driven startup is “a race, where the startup is trying to get distribution before the incumbent gets innovation”.

Please listen to that podcast episode about Dominos thinking of itself as a tech company that happens to deliver pizzas. From another episode from that same podcast, this gem of an appropriate example:

In February 2013, Ted Sarandos, Netflix’s chief content officer, told “GQ,” “the goal is to become HBO faster than HBO can become us.”

As time passes, I am increasingly skeptical that most incumbents can adapt. The culture shift is just too hard. Great software people tend to not want to work at an incumbent where the culture is not optimized to them, where they are not in charge. It is proving easier in many cases to just start a new company than try to retrofit an incumbent. I used to think time would ameliorate this, as the world adapts to software, but the pattern seems to be intensifying.

I hope he is wrong, for my sake, and for the sake of my alma mater, which is where I have chosen to work. But, um, I increasingly fear that he’s (surprise, surprise) right. Introducing technology has been hard in my workplace, but the fault lies with the culture of the workplace, not with the technology.

But as I pointed out in yesterday’s post, the regulatory capture and the cultural conformity of the higher education space in India means that most students (and their parents, or should it be the other way around) still prefer a “top” college.

A good test for how seriously an incumbent is taking software is the percent of the top 100 executives and managers with computer science degrees. For a typical tech startup, the answer might be 50-70%. For a typical incumbent, the answer may be more like 5-7%. This is a huge gap in software knowledge and skill, and you see it play out every day across many industries.

Incumbents in higher education in India – the percentage of folks with computer science degrees? Let’s move on.

First, COVID is the ultimate cover for restructuring — what my friend and former CFO Peter Currie used to call “shake and bake”. It’s an opportunity for every CEO to do all the things he/she may have wanted to do in the past to increase efficiency and effectiveness — from fundamental headcount resizing and reorganization, to changing geographic footprint, to exiting stale lines of business — but couldn’t because they would cause too much disruption. The disruption is happening anyway, so you might as well do everything you’ve always wanted to do now

75% minimum attendance, or else we reserve the right to say that you haven’t learnt enough to write the semester end examination. All classes in offline mode, only. Rote memorization tests in examination halls, with no textbooks/supplementary materials allowed. Laptops/tables/smartphones may not be used in class.

Here’s my question to those of us who work in higher education in India. Do we expect all these things to come back once the pandemic is behind us, or are we having thoughtful discussions about how the post-covid higher education field will look in India?

Today, because of the pandemic, we are at an extreme end of the spectrum which describes how learning is delivered. Everybody sits at home, and listens to a lecture being delivered (at least in Indian universities, mostly synchronously).
When the pandemic ends, whenever that may be, do we swing back to the other end of the spectrum? Does everybody sit in a classroom once again, and listens to a lecture being delivered in person (and therefore synchronously)?
Or does society begin to ask if we could retain some parts of virtual classrooms? Should the semester than be, say, 60% asynchronous, with the remainder being doubt solving sessions in classroom? Or some other ratio that may work itself out over time? Should the basic organizational unit of the educational institute still be a classroom? Does an educational institute still require the same number of in person professors, still delivering the same number of lectures?
In other words, in the post-pandemic world…
How long before online learning starts to show up in the learning statistics?

And finally, Marc Andreessen’s response to Noah’s question about what advice he (Marc) would have for a young 23 year old American:

Don’t follow your passion. Seriously. Don’t follow your passion. Your passion is likely more dumb and useless than anything else. Your passion should be your hobby, not your work. Do it in your spare time.
Instead, at work, seek to contribute. Find the hottest, most vibrant part of the economy you can and figure out how you can contribute best and most. Make yourself of value to the people around you, to your customers and coworkers, and try to increase that value every day.

The last sentence in that excerpt is another way of saying that the world is a non-zero sum game.

And I couldn’t agree more!

On The History of Public Health in India

The responses will keep you busy for days, if not weeks. This tweet, and the responses to it, are an excellent argument for why Twitter is such a valuable resource for all of us.

Asking And Answering Important Questions

Shruti Rajagopalan asked a very important question on Twitter earlier this week:

I’m writing this post on Sunday evening, which is when Shruti asked this question (and you, of course, are reading it today) but so far, there haven’t been any encouraging responses to her query, save for this one:
That would be this report, and I don’t think it was recommending large purchase orders or calling out Prime Minister Modi’s incoherent vaccine policy. This is the entire paragraph on vaccination:

Vaccines: The Committee recommended that a vaccine should pass all phases of clinical trials before it is made public. Further, it recommended that the whole population should be vaccinated. In this regard, the Committee suggested that: (i) the cost of the vaccine should be subsidised for weaker sections of society, (ii) the cold-storage system across the country should be upgraded, and (iii) vaccines should be administered as per the World Health Organization’s strategic allocation approach or a multi-tiered risk-based approach.

Long story short, the answer to Shruti’s question is: nobody. None of us were prescient enough in 2020, and that is a failure on our part.

What Shruti is really asking for is this: who is India’s Alex Tabbarok?

Why do I say this? Because Professor Tabarrok was recommending/demanding large purchase orders…

We don’t want to find ourselves with a working vaccine but too little manufacturing capacity. From an economic point of view, it would make sense to install enough capacity so that everyone in the U.S. who wanted could be vaccinated within a month. Normally, new vaccines cannot be produced so quickly and in sufficient supply. Each step of the manufacturing process must be verified and tested, and inputs to the process may face their own supply chain bottlenecks. Just as shortages of swabs and reagents delayed the rollout of testing, shortages of glass vials, bioreactors or adjuvants (a substance that increases immune stimulation) may delay vaccines. For want of a vial, the vaccine could be lost. To stand a reasonable chance of having a substantial supply of vaccines in 2021, we need to plan for capacity and reinforce supply chains now.

…on the 4th of May. That is the 4th of May 2020.

He had a post praising the idea of advance market commitments (AMC’s) out in February. Again, 2020.

And while the first excerpt up above was a plan for the USA alone, he and his collaborators expanded upon this plan, outlining what a globally coordinated plan may have looked like:

I’ve been working with Michael Kremer, Susan Athey, Chris Snyder and others to design incentives to speed vaccines and other health technologies. AcceleratingHT is our website and now features a detailed set of slides which explain the calculations behind our global plan. The global plan is similar in style to the US plan although on a larger scale. The key idea is that the global economy is losing $350 billion a month so speed pays. One way to speed a vaccine is to invest in capacity for 15-20 vaccine candidates before any candidates are approved, so that the moment a candidate is approved we can begin production (one can store doses in advance of approval). Most of the capacity will be wasted but that is a price worth paying. As Larry Summer says if you will die of starvation if you don’t get a pizza in two hours, order 5 pizzas. Human challenge trials are another way to speed the process.
A global plan is ideal since there are significant benefits to coordination. If each country invests in vaccines independently they will each choose the vaccine candidates most likely to succeed but that means all our eggs are a few baskets. There are over 100 vaccine candidates and they have different scientific and production risks so you want to choose the 15-20 which maximize the probability of success for the portfolio as a whole. To do that efficiently you need countries to agree that ‘I will invest in lots of capacity (more than I need) in candidate X if you invest in lots of capacity (more than you need) for candidate Y’, even knowing that the probability that X succeeds may be less than that of Y.

The website AcceleratingHT provides many more instances that reinforce my point, and as a student, reading the material there is genuinely useful.

A while ago, I wrote a post for students who want to work in the field of public policy. Alex Tabbarok’s work this past year is a great example of what that advice might look like in practice.

I do not know who India’s Alex Tabbarok is in 2021 – there may not even be one. But as a student, the correct question to ask is this:

What do I need to do to acquire the ability to be ahead of the curve when the next crisis comes around?

Here is my list in response to that question:

  1. Read, and write. Everyday, read and write. If you are a student of the humanities (and if you think about it, who isn’t?), you should be reading and writing everyday. It compounds, trust me.Don’t be afraid
  2. Learn the art of working backwards from the solution you want to get to. In this specific case, if you want the world to be vaccinated by the end of 2021 (let’s say), then begin by asking yourself what needs to be done to get there, but in reverse.
    7 billion vaccines will be needed – which are the manufacturers that are most likely to supply them – what do they need to get the job done – how can we get them what we need – what are the regulatory, financial, supply-chain-related hurdles they will face – how can these be removed – and so on…
    My point here is not the specifics of the exercise, whether in the case of vaccines today or something else tomorrow. My point is to learn and apply the art of working backwards from where you want to eventually be. I don’t know what you’re supposed to call this in consultant/management speak, but for starters, read about the game 21 flags in The Art of Strategy.
  3. Learn the art of being unafraid to ask big picture questions. Whenever you get that feeling of “Surely somebody somewhere must have thought to ask this question already?” – especially if you have been serious about pt. 1 above – ask the question. Repeatedly, furiously and publicly.
  4. Consume as much content as you can about crisis management from the past. (I’m working on this for my own self, and recommendations are welcome)
  5. Do not be afraid of putting out your potential solution out there. Your worst case scenario is that it is a wrong solution. As a society, we’re still better off rejecting wrong solutions than waiting for the perfect one. For rejecting a solution as being the wrong one forces us to learn more about the problem at hand.
  6. Most difficult of all: once you have offered a solution, remember that your job is to solve the original problem. Your job is to not defend your solution at all costs. This is hard.

Correlation, Causation and Thinking Things Through

Us teaching type folks love to say that correlation isn’t causation. As with most things in life, the trouble starts when you try to decipher what this means, exactly. Wikipedia has an entire article devoted to the phrase, and it has occupied space in some of the most brilliant minds that have ever been around.

Simply put, here’s a way to think about it: not everything that is correlated is necessarily going to imply causation.

For example, this one chart from this magnificent website (and please, do take a look at all the charts):

But if there is causation involved, there will definitely be correlation. In academic speak, if x and y are correlated, we cannot necessarily say that x causes y. But if x does indeed cause y, x and y will definitely be correlated.

OK, you might be saying right now. So what?

Well, how about using this to figure out what ingredients were being used to make nuclear bombs? Say the government would like to keep the recipe (and the ingredients) for the nuclear bomb a secret. But what if you decide to take a look at the stock market data? What if you try to see if there is an increase in the stock price of firms that make the ingredients likely to be used in a nuclear bomb?

If the stuff that your firm produces (call this x) is in high demand, your firm’s stock price will go up (call this y). If y has gone up, it (almost certainly) will be because of x going up. So if I can check if y has gone up, I can assume that x will be up, and hey, I can figure out the ingredients for a nuclear bomb.

Sounds outlandish? Try this on for size:

Realizing that positive developments in the testing and mass production of the two-stage thermonuclear (hydrogen) bomb would boost future cash flows and thus market capitalizations of the relevant companies, Alchian used stock prices of publicly traded industrial corporations to infer the secret fuel component in the device in a paper titled “The Stock Market Speaks.” Alchian (2000) relates the story in an interview:
We knew they were developing this H-bomb, but we wanted to know, what’s in it? What’s the fissile material? Well there’s thorium, thallium, beryllium, and something else, and we asked Herman Kahn and he said, ‘Can’t tell you’… I said, ‘I’ll find out’, so I went down to the RAND library and had them get for me the US Government’s Dept. of Commerce Yearbook which has items on every industry by product, so I went through and looked up thorium, who makes it, looked up beryllium, who makes it, looked them all up, took me about 10 minutes to do it, and got them. There were about five companies, five of these things, and then I called Dean Witter… they had the names of the companies also making these things, ‘Look up for me the price of these companies…’ and here were these four or five stocks going like this, and then about, I think it was September, this was now around October, one of them started to go like that, from $2 to around $10, the rest were going like this, so I thought ‘Well, that’s interesting’… I wrote it up and distributed it around the social science group the next day. I got a phone call from the head of RAND calling me in, nice guy, knew him well, he said ‘Armen, we’ve got to suppress this’… I said ‘Yes, sir’, and I took it and put it away, and that was the first event study. Anyway, it made my reputation among a lot of the engineers at RAND.

I learnt about this while reading Navin Kabra’s Twitter round-up from yesterday. Navin also mentions the discovery of Neptune using the same underlying principle, and then asks this question:

Do you know other, more recent examples of people deducing important information by guessing from correlated data?

… and I was reminded of this tweet:

Whether it is Neptune, the nuclear bomb or the under-reporting of Covid deaths, the lesson for you as a student of economics is this: when you marry the ability to connect the dots with the ability to understand and apply statistics, truly remarkable things can happen.

Of course, the reverse is equally true, and perhaps even more important. When you marry the ability to connect the dots with a misplaced ability to understand and apply statistics, truly horrific things can happen.

Tread carefully when it comes to statistics!

Forecasting The Future

All forecasting models are fun to learn about, and to tinker with in your software of choice. But it is equally true that all forecasting models are problematic.

First, they’re based on the assumption that the future will look like the past. Eventually, that will not be the case – this is a guarantee.

Second, even if they are based on the past, there is the problem of survivorship bias to consider in your sample of choice (my thanks to Aadisht for helping me realize this better).

And third, your predictions cannot – I repeat, cannot – account for all the underlying complexities. Forecasting is a ridiculously risky thing to do, and kudos to those who try, for this very reason.

I’d done a round-up of posts I had read in January 2020 (remember January 2020? Those were the days) that tried to predict what the world would look like when it came to India, technology and the world. I bring this up to re-emphasize the point I was trying to make in the previous paragraph: no matter how sophisticated your model, no matter how careful your sampling, and no matter however many dots you connect: reality will always have you beat.

That’s just how it is. Forecasting models work well until they don’t, and that one time they don’t can often be more costly than all the times they did.

And that brings me to this tweet:

What should you take away from this tweet (and the rest of the thread)?

My primary audience when I write here is, in a sense, myself back when I was an undergrad/post-grad student. So what advice would I want to give to myself after having read that Twitter thread?

  1. As Nitin Pai himself goes on to say in a subsequent tweet, this is a useful principle to have: Don’t try to predict the future.
  2. Respect skin in the game. Did he get it wrong? Sure he did. But hey, it takes courage to put your reasoning, your thoughts and your conclusions in the public domain. Feel free to disagree with the conclusions, but accord people who write in public the respect they deserve for having done so.
  3. Have the courage to admit you were wrong. We have two examples in front of us. One is the usual “I was misquoted/misunderstood” weasel talk. The other is an admission of error, straight up, and without qualifiers. Like the tweet above.
  4. Work at getting better. A publicly available record of your thoughts is invaluable, because it forces you to write after thinking carefully. It is also invaluable because you can outsource the “where can I get better” to the internet. And there are enough (trust me) people on the internet who will enthusiastically point out where you’re wrong. Use that advice constructively. By that I mean this, specifically: continue to write in the public domain, and that will mean making mistakes. Try not to make the same ones twice.

Like Nitin, I have written about what we’ve been going through, and how we might get out of it. All of it is available here on this blog. Some of it might turn out to be wrong – in fact, there’s a guarantee that if I write enough, some of it will be wrong. And given the pandemic that we’re going through, the stakes are impossibly high.

But it is the process of writing in public, and giving feedback on what other people write in public that drives our thinking forward.

So again, if you’re a student reading this: write. Write in the public domain. Make mistakes. Develop a thick enough skin to take on the criticism. Learn the (almost impossible to acquire) skill of figuring out when you’re wrong, and develop and hone the courage it takes to admit it.

And then, write again.

(Quick note: posting will be sporadic for some time.)

Help Me Understand This, Somebody…

A fellow Puneri citizen sent out this tweet yesterday:

It was hard not to be snarky, and I didn’t even bother trying to resist:

But in my day job, I try to be an economist, and so I have questions. Just two of them, and they’re fairly simple ones. Here they are:

  1. He (or SII) was free to set the price, correct? Free market economics: let the seller decide the price, and let the buyer decide if she wants to buy at that price.

    So the price now stands reduced by a whopping 25%. Does that mean that it was set too high in the first place?

    That is, let us assume that SII is able to increase capacity expansion at a price of 300 per dose. Also assume that it can make a normal or “super” profit at this price – then was 400 not too high?

    If we assume that he was going to earn an extra 100 rupees per vaccine sold, and that he was going to sell say 200 million vaccines to the states, that’s 200 million into 100 rupees.1

    I don’t want to do the math, but were we ok with at least that much “extra” money going into the Poonawalla coffers until yesterday?

    If yes, why?

  2. Unless, of course, that was not the case, and capacity expansion will suffer at a price of 300. A raise in the minimum wage will mean switched-off air-conditioning, correct? Well, in that case, is it not our moral duty to ask him to take the price back up to 400? Because if the opportunity cost of his philanthropy is reduced capacity expansion, isn’t that worse?

(By the way, all this is taking the assumption that SII “needs” the proceeds from the sale of this one vaccine alone to fund capacity expansion. That may or may not be true. And this also assumes that this is the only vaccine that SII will be producing and selling, which is obviously not true. Even in this “best-case” scenario, my questions hold up – if we do a full reckoning, they become even more important!)

If it is the first point above, us economists must explain why we think it is ok for those 100 rupees (per dose) to go into SII’s coffers.

If it is the latter, there ought to be a stream of op-eds beseeching Mr. Poonawalla to roll back his offer, for that would be truly philanthropic.

Which will it be?

And I know I said only two questions, but forgive me my greed, and let me ask one more: what is the definition of “transparent pricing”?

  1. Where do I get that number 200 million from? Who knows? I assumed that for the 960 million people in total who become/continue to be eligible on the 1st of May, he gets to sell only 200 million doses to the states. And yes, I am assuming only a single dose for these 200 million. Since nobody knows what the quantities are actually going to be, this is as reasonable an assumption as any other. If anything, this is a very conservative estimate. No?[]

The Vaccine Responsibility by K. Sujatha Rao

I and a friend have been exchanging messages about the idiocy that was the European Super League (or whatever the name was. I’m not even going to bother looking it up). He asked me if I would post anything here about the economics behind the league, either defending the idea or refuting it.

Here’s one paragraph from a previous post of mine that I would want to base that essay (if I ever write it) on:

When David Perell says that we have made the world cheaper, what I think he is saying is that we have figured out ways to cheapen the effort that we are willing to put into the act of consuming something. That something could be a meal, but it could also be extended to reading, viewing, or listening as well – and more besides.

To me, the ESL is based on the assumption that a large number of us fans are willing to cheapen the effort that we are willing to put into the act of consuming football matches – that we are not looking to maximize soul. Thankfully, so far, that assumption seems to have been ever-so-slightly off the mark.1

Anyways, he wrote a post about it, and we’ve been sharing stuff about it on and off. I sent him one written by Jonathan Wilson, and he sent me this response: “I seem to have covered largely the same points with way less finesse.”

To which my response was, that is my middle name.

Now, I didn’t need to write any of that, and any half-decent editor would have lopped off that whole bit – and quite rightly too. But hey, this is my blog, I had fun writing it, and it was a welcome break from you-know-what.

But “I seem to have covered largely the same points with way less finesse” is all too applicable in my case when reading K. Sujatha Rao’s excellent article in the Indian Express today on India’s vaccination drive. It is worse in my case of course, because I haven’t even covered largely the same points, forget the finesse.

I won’t excerpt from it, because it deserves to be read in its entirety.

But I will reiterate (in my won words) points that I thought were especially crucial, and also list out some questions that I had after I finished reading it.

  1. The point about the utilization of the vaccines that will be procured by the Government of India (GoI) is really a request for clearer communication. You simply cannot overstate the importance of clear communication.
  2. If SII has received grants from GAVI and from the Central government, would a publicly available dashboard about capacity, supply chain bottlenecks, vaccine allocation by states/countries and specific timelines for capacity expansion be possible? I’m not trying to be snarky, I am genuinely asking. What are the reasons against such a dashboard being made publicly available?
  3. I’ll ask again: Imagine a good with large positive externalities. Let’s call this good a vaccine. Let’s say that the supply of these vaccines is going to be tight. Let’s say that demand is very inelastic. Would you recommend fragmenting total market demand into smaller constituent curves? If yes, why?
    K. Sujatha Rao makes the legal argument in her article, and then makes the economic argument. Read both (and again, not being snarky), please tell me why we are choosing to do what we are doing – in this year of all years, and for this good of all goods.
  4. She speaks of halving the estimated cost of INR 60,000 crores by invoking compulsory licensing and expanding production through the 18 manufacturing facilities in India. These must be back of the envelope calculations, obviously. But if anybody reading this has information on how one might create such an estimate (and an estimate of how much production will go up by via this route), please let me know.

Please, do read the article.

  1. Sarcasm alert: the evil idiots who dreamt up that monstrosity have gotten exactly what they deserved. Not enough of it, if you ask me.[]