A Sunny Outlook

Some years ago, I wrote a chapter in a book called Farming Futures. The book is about social entrepreneurship in India, and my chapter was about a firm called Skymet. Skymet is a private weather forecasting firm based partially out of Pune and partially out of Noida (along with other office in other locations). But researching for the chapter got me interested in both how the art and science of weather forecasting had developed over time, and where it is headed next.

Only trivia enthusiasts are likely to remember the name of the captain on whose ship Charles Darwin made his historic voyage that was to result in the publication of “On the Origin of Species”. Fewer still will remember that Admiral Robert FitzRoy committed suicide. The true tragedy, however, is that it is almost certainly his lifelong dedication to predicting the weather that caused him to take his own life.
We have, in the decades and centuries since, come a long way. Weather forecasting today is far more advanced than it was in Admiral FitzRoy’s day. Britain, for example, Admiral FitzRoy’s own nation, today has an annual budget of more than 80 million GBP to run its meteorological department. It has an accuracy of around 95% when it comes to forecasting temperatures, and an accuracy of around 75% when it comes to forecasting rain – anybody who is even remotely familiar with Britain’s notoriously fickle weather would know that this is no small achievement.

Farming Futures: Emerging Social Enterprises in India

Those numbers that I cited, and the tragic story of Admiral FitzRoy, come from a lovely book called The Weather Experiment.


But I first read about weather, and the difficulties associated with forecasting it in a book called Chaos, by James Gleick:

Lorenz enjoyed weather—by no means a prerequisite for a research meteorologist. He savored its changeability. He appreciated the patterns that come and go in the atmosphere, families of eddies and cyclones, always obeying mathematical rules, yet never repeating themselves. When he looked at clouds, he thought he saw a kind of structure in them. Once he had feared that studying the science of weather would be like prying a jack-in–the-box apart with a screwdriver. Now he wondered whether science would
be able to penetrate the magic at all. Weather had a flavor that could not be expressed by talking about averages. The daily high temperature in Cambridge, Massachusetts, averages 75 degrees in June. The number of rainy days in Riyadh, Saudi Arabia, averages ten a year. Those were statistics. The essence was the way patterns in the atmosphere changed over time…

Ch. 1, The Butterfly Effect, Chaos, by James Gleick

What is the Butterfly Effect, you ask? It gets its own Wikipedia article, have fun reading it.


All of which is a very long way to get around to the write-up we’re going to be talking about today, called After The Storm.

On 29 October 1999, a “Super Cyclone” called Paradip devastated parts of Odisha and the east coast of India. At wind speeds of almost 250 kms per hour, it ravaged through the land, clearing out everything in its path. Fields were left barren, trees uprooted like mere matchsticks, entire towns devastated. More than 10,000 people lost their lives.
Fast forward to two decades later. In 2020, bang in the middle of the Covid-19 pandemic, another cyclone—known as Amphan—speeds through the Bay of Bengal. It crashes into the land like Paradip did in 1999. Like before, many homes are destroyed and structures uprooted. But one thing is different: this time’s death toll is 98. That’s a 100 times lower than 1999’s casualties.
What made this difference possible? Simply put: better, timely and more accurate weather prediction.

https://fiftytwo.in/paradigm-shift/after-the-storm/

We’ve made remarkable progress since the days of Admiral FitzRoy. Predicting the weather is still, admittedly, a very difficult and very expensive thing, as this lovely little write-up makes clear, but it is also something we’re much better at these days. We have better instruments, better computing power, better mathematical and statistical tools to deploy, and the ability to synthesize all of these to come up with much better forecasts – but it’s not perfect, and it’s not, well, good enough.

Those last two words aren’t meant as a criticism or a slight – far from it. The meteorologists themselves feel that is is not good enough:

“It almost becomes like flipping a coin,” Professor Islam says. “The IMD is not to be blamed. They will be very good at predicting the weather three or four days in advance. Beyond that, it cannot be done because there is a fundamental mathematical limitation to these questions.”
“IMD can do another sensor, another satellite, they can maybe improve predictions from two days, to three days. But can they do ten days? There is no evidence. Right now there is no weather forecasting model on the globe. India to Europe to Australia, it doesn’t matter, it’s not there.”

https://fiftytwo.in/paradigm-shift/after-the-storm/

As Professor Islam says, he wants to move from up from being able to forecast the next four to five days, to being able to predict weather over the next ten days. Why? So that communities in the path of a storm have adequate time to move. What could be more important than that when it comes to meteorology.


So what’s the constraint? This is a lovely analogy:

“I give this example to my students,” the professor says, “Look, usually all of science and AI is based on this idea of driving with the rearview mirror. I don’t have an option, so I’m looking into my rearview mirror and driving. I will be fine as long as the road in the front exactly mirrors the rearview. If it doesn’t and I go into a turn? Disastrous accident.”

https://fiftytwo.in/paradigm-shift/after-the-storm/

It’s weird what the human brain will choose to remind you of, but this reminds me, of all things, of a gorilla. That too, a gorilla from a science fiction book:

Amy distinguished past, present, and future—she remembered previous events, and anticipated future promises—but the Project Amy staff had never succeeded in teaching her exact differentiations. She did not, for example, distinguish yesterday from the day before. Whether this reflected a failing in teaching methods or an innate feature of Amy’s conceptual world was an open question. (There was evidence for a conceptual difference.) Amy was particularly perplexed by spatial metaphors for time, such as “that’s behind us” or “that’s coming up.” Her trainers conceived of the past as behind them and the future ahead. But Amy’s behavior seemed to indicate that she conceived of the past as in front of her—because she could see it—and the future behind her— because it was still invisible.

Michael Crichton, Congo

That makes a lot of sense, doesn’t it? And that’s the fundamental problem with any forecasting tool: it necessarily has to be based on what happened in the past, because what else have we got to work with?

And if, as Professor Islam says, the road in the future isn’t exactly like the past, disaster lies ahead.


But Artificial Intelligence and Machine Learning need not be about predicting what forms the storms of the future might take. They can be of help in other ways too!

“It hit us that the damage that happened to the buildings in the poorer communities could have been anticipated very precisely at each building’s level,” Sharma explains. “We could have told in advance which roofs would fly away, and which walls would collapse, which not so. So that’s something we’ve tried to bring into the AI model, so that it can be a predictive model.”

“What we do is, essentially, this: we use satellite imagery or drone imagery and through that, we identify buildings. We identify the material and technology of the building through their roofs as a proxy, and then we simulate a sort of a risk assessment of that particular building, right? We also take the neighbouring context into account. Water bodies, how high or low the land is, what kind of trees are around it, what other buildings are around it.”

The team at SEEDS and many others like it are more concerned about the micro-impact that weather events will have. Sharma is interested in the specifics of how long a building made from a certain material will be able to withstand the force of a cyclone. This is an advanced level of interpretation we’re talking about. It’s creative, important and life-saving as well.

https://fiftytwo.in/paradigm-shift/after-the-storm/

In other words, we may not know the intensity of a particular storm, and exactly when and where it will hit. But given assumptions of the intensity of a storm, can we predict which buildings will be able to withstand a given storm and which ones won’t?

This is, as a friend of mine to whom I forwarded this little snippet said, is very cool.

I agree. Very cool indeed.

And sure, accuracy about weather forecasting may still be a ways away, and may perhaps lie forever beyond our abilities. But science, mathematics and statistics might still be able to help us in other ways, and that (to me) still counts as progress.

And that is why, all things considered, I’d say that when it comes to the future of weather forecasting, sunny days are ahead.


In case you haven’t already, please do subscribe to fiftytwo.in

Excellent, excellent stories, and the one I have covered today is also available in podcast form, narrated by Harsha Bhogle, no less. All their other stories are wroth reading too, and I hope you have as much fun going through them as I have.

Lessons from the eradication of smallpox

Vox has a nice and short read out on the battle against smallpox, and lessons we might learn today from how and where the battle was waged, at what costs, and with what effects.

But for all that the world has lost in the last few years, the history of infectious disease has a grim message: It could have been even worse. That appalling death toll resulted even though the coronavirus kills only about 0.7 percent of the people it infects. Imagine instead that it killed 30 percent — and that it would take centuries, instead of months, to develop a vaccine against it. And imagine that instead of being deadliest in the elderly, it was deadliest for young children.
That’s smallpox.

https://www.vox.com/future-perfect/21493812/smallpox-eradication-vaccines-infectious-disease-covid-19

My notes after having read the article:

  1. Smallpox is estimated to have killed between 300 million to 500 million people in the 20th century alone
  2. We still do not have an effective treatment against smallpox
  3. There are two different viruses that cause smallpox: variola major and variola minor
  4. We no longer need to explain R0 to anybody, thanks to covid, but this point is staggering: it had an infectiousness of between 5 to 7, and a mortality rate of 30%.
  5. “In China, as early as the 15th century, healthy people deliberately breathed smallpox scabs through their noses and contracted a milder version of the disease. Between 0.5 percent and 2 percent died from such self-inoculation, but this represented a significant improvement on the 30 percent mortality rate of the disease itself.”
    What a horrible lottery to play. Would you play this lottery? This, by the way, is one of the many reasons why learning statistics and probability is worth your time.
  6. Learn more about Edward Jenner.
  7. We have better ways of shipping vaccines across the world these days, but what a story this is!
    “Spain especially struggled to reach its colonies in Central and South America, so in 1803, health officials in the country devised a radical new method for distributing the vaccine abroad: orphan boys.
    The plan involved putting two dozen Spanish orphans on a ship. Right before they left for the colonies, a doctor would give two of them cowpox. After nine or 10 days at sea, the sores on their arms would be nice and ripe. A team of doctors onboard would lance the sores, and scratch the fluid into the arms of two more boys. Nine or 10 days later, once those boys developed sores, a third pair would receive fluid, and so on. (The boys were infected in pairs as backup, just in case one’s sore broke too soon.) Overall, with good management and a bit of luck, the ship would arrive in the Americas when the last pair of orphans still had sores to lance. The doctors could then hop off the ship and start vaccinating people.”
  8. Institutions matter:
    “It was not until the 1950s that a truly global eradication effort began to appear within reach, thanks to new postwar international institutions. The World Health Organization (WHO), founded in 1948, led the charge and provided a framework for countries that were not always on friendly terms to collaborate on global health efforts.”
  9. Culture matters:
    “Efforts by the British Empire to conduct a smallpox vaccination program in India made less progress, due in large part to mistrust by the locals of the colonial government.”
  10. Science matters:
    ” “There was no shortage of people telling [the people involved in the eradication effort] that their effort was futile and they were hurting their career chances,” former CDC director William Foege wrote in his 2011 book House on Fire about the smallpox eradication effort.
    But other advances had brought it within reach. Needle technology had improved, with new bifurcated needles making it possible to use less vaccine. Overseas travel improved, which made it easier to ship vaccines and get public health workers where they were most needed, and provided impetus for worldwide eradication as it made it more likely that a smallpox outbreak anywhere in the world could spread.”

As always, read the whole article. I’ll quote here the concluding paragraph from the piece, and I’d urge you to reflect on it:

The devastation of Covid-19 has hopefully made us aware of the work public health experts and epidemiologists do, the crucial role of worldwide coordination and disease surveillance programs (which are still underfunded), and the horrors that diseases can wreak when we can’t control them.
We have to do better. The history of the fight against smallpox proves that we’re capable of it.

https://www.vox.com/future-perfect/21493812/smallpox-eradication-vaccines-infectious-disease-covid-19

Futurology from 1967

Did no work of science fiction/futurology anticipate miniaturization? Genuine question.

AI/ML: Some Thoughts

This is a true story, but I’ll (of course) anonymize the name of the educational institute and the student concerned:

One of the semester end examinations conducted during the pandemic at an educational institute had an error. Students asked about the error, and since the professor who had designed the paper was not available, another professor was asked what could be done. Said professor copied the text of the question and searched for it online, in the hope that the question (or a variant thereof) had been sourced online.

Alas, that didn’t work, but a related discovery was made. A student writing that same question paper had copied the question, and put it up for folks online to solve. It hadn’t been solved yet, but the fact that all of this could happen so quickly was mind-boggling.

The kicker? The student in question had not bothered to remain anonymous. Their name had been appended with the question.

Welcome to learning and examinations in the time of Coviid-19.


I have often joked in my classes in this past decade that it is only a matter of time before professors outsource the design of the question paper to freelance websites online – and students outsource the writing of the submission online. And who knows, it may end up being the same freelancer doing both of these “projects”.

All of which is a very roundabout way to get to thinking about Elicit, videos about which I had put up yesterday.

But let’s begin at the beginning: what is Elicit?

Elicit is a GPT-3 powered research assistant. Elicit helps you classify datasets, brainstorm research questions, and search through publications.

https://www.google.com/search?q=what+is+elicit.org

Which of course begs a follow-up question: what is GPT-3? And if you haven’t discovered GPT-3 yet, well, buckle up for the ride:

GPT-3 belongs to a category of deep learning known as a large language model, a complex neural net that has been trained on a titanic data set of text: in GPT-3’s case, roughly 700 gigabytes of data drawn from across the web, including Wikipedia, supplemented with a large collection of text from digitized books. GPT-3 is the most celebrated of the large language models, and the most publicly available, but Google, Meta (formerly known as Facebook) and DeepMind have all developed their own L.L.M.s in recent years. Advances in computational power — and new mathematical techniques — have enabled L.L.M.s of GPT-3’s vintage to ingest far larger data sets than their predecessors, and employ much deeper layers of artificial neurons for their training.
Chances are you have already interacted with a large language model if you’ve ever used an application — like Gmail — that includes an autocomplete feature, gently prompting you with the word ‘‘attend’’ after you type the sentence ‘‘Sadly I won’t be able to….’’ But autocomplete is only the most rudimentary expression of what software like GPT-3 is capable of. It turns out that with enough training data and sufficiently deep neural nets, large language models can display remarkable skill if you ask them not just to fill in the missing word, but also to continue on writing whole paragraphs in the style of the initial prompt.

https://www.nytimes.com/2022/04/15/magazine/ai-language.html

It’s wild, there’s no other way to put it:


So, OK, cool tech. But cool tech without the ability to apply it is less than half of the story. So what might be some applications of GPT-3?

A few months after GPT-3 went online, the OpenAI team discovered that the neural net had developed surprisingly effective skills at writing computer software, even though the training data had not deliberately included examples of code. It turned out that the web is filled with countless pages that include examples of computer programming, accompanied by descriptions of what the code is designed to do; from those elemental clues, GPT-3 effectively taught itself how to program. (OpenAI refined those embryonic coding skills with more targeted training, and now offers an interface called Codex that generates structured code in a dozen programming languages in response to natural-language instructions.)

https://www.nytimes.com/2022/04/15/magazine/ai-language.html

For example:

(Before we proceed, assuming it is not behind a paywall, please read the entire article from the NYT.)


But about a week ago or so, I first heard about Elicit.org:

Watch the video, play around with the tool once you register (it’s free) and if you are at all involved with academia, reflect on how much has changed, and how much more is likely to change in the time to come.

But there are things to worry about, of course. An excellent place to begin is with this essay by Emily M. Blender, on Medium. It’s a great essay, and deserves to be read in full. Here’s one relevant extract:

There is a talk I’ve given a couple of times now (first at the University of Edinburgh in August 2021) titled “Meaning making with artificial interlocutors and risks of language technology”. I end that talk by reminding the audience to not be too impressed, and to remember:
Just because that text seems coherent doesn’t mean the model behind it has understood anything or is trustworthy
Just because that answer was correct doesn’t mean the next one will be
When a computer seems to “speak our language”, we’re actually the ones doing all of the work

https://medium.com/@emilymenonbender/on-nyt-magazine-on-ai-resist-the-urge-to-be-impressed-3d92fd9a0edd

I haven’t seen the talk at the University of Edinburgh referred to in the extract, but it’s on my to-watch list. Here is the link, if you’re interested.

And here’s a Twitter thread by Emily M. Blender about Elicit.org specifically:


In response to this critique and other feedback, Elicit.org have come up with an explainer of sorts about how to use Elicit.org responsibly:

https://ought.org/updates/2022-04-25-responsibility

Before we proceed, I hope aficionados of statistics have noted the null hypothesis problem (which error would you rather avoid) in the last sentence of pt. 1 in that clipping above!


So all that being said, what do I think about GPT3 in general and elicit.org in particular?

I’m a sucker for trying out new things, especially from the world of tech. Innocent until proven guilty is a good maxim for approaching many things in life, and to me, so also with new tech. I’m gobsmacked to see tools like GPT3 and DallE2, and their applications to new tasks is amazing to see.

But that being said, there is a lot to think about, be wary of and guard against. I’m happy to keep an open mind and try these amazing technologies out, while keeping a close eye on what thoughtful critics have to say.

Which is exactly what I plan to do!

And for a person with a plan such as mine, what a time to be alive, no?

Have you tried Elicit.org yet?

Video 1:

And Video 2:

The Case For Doubling Spending on R&D

Timothy Taylor, author of the blog The Conversable Economist, has a nice post out on the case for doubling R&D spending. He speaks of doubling spending on R&D by the US government, but the point is equally applicable to all governments, including India’s.

The post is a reflection on a chapter in an e-book published by the Aspen Group. The chapter has been written by Benjamin F. Jones, and is titled “Science and Innovation: The Under-Fueled Engine of Prosperity.” (pp. 272 in the PDF that has been linked to above). Timothy Taylor shares an extract that ought to familiar to us in terms of the direction in which scientific progress has been headed, and perhaps even the magnitude – but every now and then, it helps to remind ourselves how far we’ve come:

Real income per-capita in the United States is 18 times larger today than it was in 1870 (Jones 2016). These gains follow from massive increases in productivity. For example, U.S. corn farmers produce 12 times the farm output per hour since just 1950 (Fuglie et al. 2007; USDA 2020). Better biology (seeds, genetic engineering), chemistry (fertilizers, pesticides), and machinery (tractors, combine harvesters) have revolutionized agricultural productivity (Alston and Pardey 2021), to the point that in 2018 a single combine harvester, operating on a farm in Illinois, harvested 3.5 million pounds of corn in just 12 hours (CLASS, n.d.). In 1850, it took five months in a covered wagon to travel west from Missouri to Oregon and California, but today it can be done in five hours—traveling seven miles up in the sky. Today, people carry smartphones that are computationally more powerful than a 1980s-era Cray II supercomputer, allowing an array of previously hard-to-imagine things—such as conducting a video call with distant family members while riding in the back of a car that was hailed using GPS satellites overhead.

https://conversableeconomist.com/2022/04/19/the-case-for-doubling-us-rd-spending/

The latter part of the extract, which I’ve not quoted here, is about the increase in life expectancy, and is also worth reading. Post the extract, Timothy Taylor goes on to speak about how it is important to celebrate the fact that we were able to push out vaccines in the space of a little less than a year, which is a stellar achievement. And indeed it is! You might have differing opinions about the efficacy of these vaccines, and you might even be of the opinion that the firms doth profit too much from their creation, but I hope you agree that the fact that we were able to do this at all, and as rapidly as we did, is testimony to have far we have come as a civilization.

As an aside, read also this Washington Post editorial about the discovery of the virus, and how the message didn’t get out nearly quickly enough (duh.)

Both points are important to understand as students. Which two points, you ask? That progress as a civilization depends on two things: the rate of technological progress, and the underlying culture that enables it, embraces it and uses it properly. For reading the editorial, I came away with the opinion that China had the technology, but lacked the culture.

I would urge you to think about how this might resonate with each of us as individuals: we have the technology to be ever more productive, and the technology improves every year. But have we built for ourselves a culture of allowing ourselves to use this technology as efficiently as we should? What about the institutions that each of us work for or study in? What about the countries we stay in? Technological progress without an enabling culture doesn’t work, and as students of productivity (that’s one way to think about studying economics), you need to be students of both aspects.


Anyway, back to scientific progress. One of the points that Jones makes in his chapter is that the US has been lagging behind the current leaders on two different metrics: total R&D expenditure as a percentage of GDP, and public R&D expenditure as a share of GDP. China’s R&D expenditure has seen an annual increase of 16% since the year 2000, while the US is at 3% annual growth.

What about India, you ask? Here’s a chart from an Indian Express article about the topic:

https://indianexpress.com/article/opinion/columns/unesco-stats-on-global-expenditure-on-r-d-7775626/

As the article points out, let alone trying to compute the rate of increase, we actually seem to be on a downward trajectory for a metric called GERD, which stands for Gross Domestic Expenditure on Research and Development. Here’s the link to the data from the World Bank.

https://data.worldbank.org/indicator/GB.XPD.RSDV.GD.ZS?end=2018&locations=IN&start=2000&view=chart

We clearly need to do better. That article in the Indian Express ends with this paragraph:

A commitment from the Centre to raise GERD to 1 per cent of the GDP in the next three years could be one of the most consequential decisions taken in the 75th year of India’s independence.

https://indianexpress.com/article/opinion/columns/unesco-stats-on-global-expenditure-on-r-d-7775626/

And that is a nice segue back to the blog post that we started today’s post with. If you’re asking (and I hope you are!) questions along the lines of why it should be the government and not the private sector, I have two answers for you. One, the truth always lies somewhere in the middle, and so you need both private and government spending. And two, there is an economic argument for your consideration:

Jones’s essay reviews the argument, fairly standard among economists, that a pure free market will tend to underinvest in new technologies, because in a pure free market the innovator will not capture the full value of an innovation. Indeed, if firms face a situation where unsuccessful attempts at innovation just lose money, while successful innovations are readily copied by others, or the underlying ideas of the innovation just lead to related breakthroughs for others, then the incentives to innovate can become rather thin, indeed. This is the economic rationale for government policies to support research and development: direct support of basic research (where the commercial applications can be quite unclear), protection of intellectual property like patents and trade secrets, tax breaks for companies that spend money on R&D, and so on.

https://conversableeconomist.com/2022/04/19/the-case-for-doubling-us-rd-spending/

Now, how much of the lifting should be done by government, and how much should be done by the private sector is a debate that will never end, but here is an EFE post that might help you start to think through the process.


Timothy Taylor and Benjamin F. Jones argue that the US needs to spend more on R&D, and that the U.S. government should do more in this regard.

My contention is two-fold: that this point applies with even more urgency in the Indian context, and that an enabling culture is an equally important concept, but an underrated one the world over.

Supply and Demand, Complements and Substitutes and Dalle-E 2

Before we begin, and in case some of you were wondering:

Early last year, San Francisco-based artificial intelligence company OpenAI launched an AI system that could generate a realistic image from the description of the scene or object and called it DALL.E. The text-to-image generator’s name was a portmanteau coined after combining the artist Salvador Dali and the robot WALL.E from the Pixar film of the same name.

https://analyticsindiamag.com/whats-the-big-deal-about-dall-e-2/

Dall-E 2 is amazing. There are ethical issues and considerations, sure, but the output form this AI system is stunning:

A rabbit detective sitting on a park bench and reading a newspaper in a Victorian setting (Source)

And just in case it isn’t clear yet, no such painting/drawing/art existed until this very sentence, the one that is the caption, was fed to the AI. And it is the AI that “created’ this image. Go through the entire thread.


This has led, as might be expected, to a lot of wondering about whether artists are going to be out of a job, and the threats of AI to humanity at large. I do not know enough to be able to offer an opinion one way or the other where the latter is concerned, but I do, as an economist, have some points to make about the former.

These thoughts were inspired by reading Ben Thompson’s latest (freely available) essay on Dall-E 2, titled “DALL-E, the Metaverse, and Zero Marginal Content“. He excerpts from the OpenAI website in his essay, and this sentence stood out:

DALL-E is an example of how imaginative humans and clever systems can work together to make new things, amplifying our creative potential.

https://openai.com/dall-e-2/

And that begs an age-old question where economists are concerned: is technology a complement to human effort, or a substitute for it? The creators of Dall-E 2 seem to agree with Steve Jobs, and think that the AI is very much a complement to human ingenuity, and not a substitute for it.

I’m not so sure myself. For example: is Coursera for Campus a complement to my teaching or a substitute for it? There are many factors that will decide the answer to this question, including quality, price and convenience among others, and complementarity today may well end up being substitutability tomorrow. If this isn’t clear, think about it this way: cars and drivers were complementary goods for decades, but today, is a self-driving car a complement or a substitute where a driver is concerned?

But for the moment, I agree: this is an exciting new way to generate content, and is likely to work best when used as a complement by artists. Note that this is based on what I’ve seen and read – I have not myself had a chance to use or play around with Dall-E 2.


The title of today’s blog post is about substitutes and complements, which we just finished talking about in the previous section, but it also includes references to demand and supply. What about demand and supply?

Well, Ben Thompson talks about ways to think about social media firms today. He asks us to think about Facebook for example, and asks us to reflect upon where the demand and the supply for Facebook as a service comes from.

Here’s my understanding, for having read Ben Thompson’s essay: Facebook’s demand comes from folks like you and I wanting to find out what, well, folks like you and I are up to. What are our friends, our neighbors, our colleagues and our acquaintances up to? What are their friends, neighbors, colleagues and acquaintances up to? That’s the demand.

What about the supply? Well, that’s what makes Facebook such a revolutionary company – or at least, made it revolutionary back then. The supply, as it turns out, also came from folks like you and I. We were (and are) each others friends, neighbors, colleagues and acquaintances. Our News Feed was mostly driven by us in terms of demand, and driven by us in terms of supply. Augmented by related stuff, and by our likes and dislikes, and news sources we follow and all that, but demand and supply comes from our own networks.

TikTok, Thompson says, is also a social network, and supply and demand is also user driven, but it’s not people like us that create supply. It is just, well, people. TikTok “learns” what kind of videos we like to see, and the algorithm is optimized for what we like to see, regardless of who has created it.

But neither Facebook nor TikTok are in the business of generating content for us to see. The former, to reiterate, shows us stuff that our network has created or liked, while the latter shows us stuff that it thinks we will like, regardless of who has created it.

But how long, Ben Thompson’s essay asks, before AI figures out how to create not just pictures, but entire videos. And when I say videos, not just deep fakes, which already exist, but eerily accurate videos with depth, walkthroughs, nuance, shifting timelines and all the rest of it.

Sounds far-fetched?

Well, I remember taking an hour to download just one song twenty years ago, and I can now stream any song in the world on demand. And soon (already?) I will be able to “create” any song that I like, by specifying mood, genre, and the kind of lyrics I want.

How long before I can ask AI to create a movie just for me? Or just me and my wife? Or a cartoon flick involving me and my daughter? How long, in other words, before my family’s demand for entertainment is created by an AI, and the supply comes from that AI being able to tap into our personal photo/video collection and make up a movie involving us as cartoon characters?

Millions of households, cosily ensconced in our homes on Saturday night, watching movies involving us in whatever scenario we like. For homework, read The Secret Life of Walter Mitty by Thurber (the short story, please, not the movie!), Snowcrash by Neal Stephenson, and The Seven Basic Plots by Baker.


There are many tantalizing questions that arise from thinking about this, and I’m sure some have struck you too. But I don’t want to get into any of them right now.

Today’s blog post has a very specific point: it doesn’t matter how complicated the issue at hand is. Simple concepts and principles can go a very long way in helping you frame the relevant questions required for analysis. Answering them won’t be easy, as in this case, but hey, asking (some of) the right questions is a great place to start.

Dall.E 2

It’s been a week or so since I’ve seen this, and I remain gobsmacked

Meet the MIT Banana Lounge

Yup, really. There’s a place in MIT where you can lounge around and eat bananas. A lot of bananas.

https://twitter.com/iaincheeseman/status/1513467068351451137

How many is a lot, you ask? 280,000 bananas in this academic year alone. This is a project run by the Undergraduate Association at MIT, and they also place pianos around campus for folks to give it a try, and for those of us who prefer a more sedate outlook towards life, they also have a hammocks team, who are doing exactly what you hope they would.

The bananas are for free, by the way. If you happen to be on the MIT campus, you can drop in and chomp away to your heart’s content, courtesy an MIT alum who’s also been known to, um, do other stuff besides.

Cool stuff, right?


The reason I bring this up is because I and a student at GIPE were chatting the other day about questions that her juniors were asking her. And the question was about how they didn’t have “enough R projects” to do. (R, for the uninitiated, is a software that econ nerds like to freak out over.)

I’m always a little befuddled when students say they don’t have projects to work on, or are looking for datasets to work on. The lazy answer to give to queries such as this is something along the lines of Kaggle, or Google’s Dataset Search. There’s hundreds of such data sources available online for free, and they’re one simple Google search away, so that’s one reason for my befuddlement.

But the primary source of my befuddlement is the fact that students in possession of a software looking for a dataset is very much a case of the cart being put in front of the horse! Software is a tool that helps you in the work you’re doing. But the approach that most students take is that they have the chops to use the software, and they don’t know what work to do.


You could always try and see if you can get an alumni to buy bananas, and forecast demand for bananas!

Trend! Seasonality! Forecasting! For bananas consumed on campus.


I ask you: which is a cooler story to tell? A story in which you say that you downloaded a dataset from the internet and did some modeling with it…

OR

A story in which you say that you and a bunch of your friends got together and convinced your college to give a room to stock bananas, convinced an alumni member to sponsor these bananas, figured out the logistics to procure, transport and store these bananas, and used a tool called R (or Python, or SAS or SPSS or whatever) to forecast demand?

The second option teaches you project management, the art of pitching a proposal, teamwork, logistics and coding. And so much more besides! It builds a story that works for the team, the institute, the community, and you use a statistical software the way it was meant to be used: as a tool that makes your life easier.

I know which story gets my vote.


You could build shared calendars, YouTube playlists using Google Sheets, demos for sampling using Google Sheets, or anything else that takes your fancy. Use Statsguru to analyze cricket stats using Python, automate the creation of book recommendation websites, or well, give bananas away for free.

But datasets for projects?

You’re limited by your imagination alone.

Why Is Reading the News Online Such a Pain?

Livemint, Hindu Business Line, Business Standard, Times of India, The New York Times, The Hindu, The Washington Post, The Economist, Bloomberg Quint and Noah Smith’s Substack.

These are, as of now, my sources of news online that I pay for.

There are other newsletters that I subscribe to and pay for (The Browser is an excellent example), and I read stuff published in other newspapers too, but I’m restricting myself to only the current news sources that I pay for. I would like to subscribe to the Financial Times and to Stratechery too, but my budget line begins to cough firmly and insistently at this point, more’s the pity.

But here’s the thing: reading news online sucks.


Some are worse than others, and I’m very much looking at you, Business Standard. Their app is a joke, and the number of times one has to sign in while reading the paper on a browser isn’t funny. Some are, relatively speaking, better. The NYT website and app are both pretty good, as is the Economist. But still, it isn’t friction free, and there really should be a way to get the user experience to be better than it is right now.

And more than better, a more urgent word is uniform. Here’s a simple use case: let’s say I want to read articles on the current lockdown in Shanghai. I have to go to each website, and either run a search, or navigate to the appropriate section. But on each website, the search button will be located in a slightly different place, with a slightly different user experience. Each website while have their own navigation system. Each website will have different ways to filter search results.

Some will allow you to copy excerpts, some won’t. Some will allow clips and force an appendage at the end (“Read More At XYZ” – I’m looking at you, ToI). But by the time I finish visiting the third website to read about the topic I wanted to – current lockdowns in Shanghai – I’m pretty much done out of sheer exasperation.


It shouldn’t be this hard!

Workarounds kind of exist. For example, I can add the RSS feeds to Feedly, or any other feed reader of your choice. If you’re not familiar with Feedly, or RSS readers in general, here is an old post about it. But the reason I say kind of is because most (if not all) newspapers will not provide the full article in the RSS feed. You have to click through to read the full thing.

Not much use, is it?

Which, to be clear, is entirely understandable. User tracking, ads, and all the rest of it, I get it. But it does mean that Feedly isn’t a great way to keep track of all these articles in one place.

What I would really like is an app/service that aggregates all news sources in full in one place, and allows me to sign in to premium news sources via that app/service.

Does such a service exist? Or are there workflows that solve this problem?

Please, do let me know!