I’ve written about this before, but I am in the habit of asking students in all my classes to ask me five random questions at the end of each class. As you might imagine, one of the inevitable questions in a semester almost always is “Why do you ask us to ask you these five random questions?”
So often have I been asked this question that I have the answer down pat, and it goes something like this:
- Because according to me, our education system prioritizes being able to answer questions perfectly, rather than frame really good questions. Think of this little exercise as a contrarian attempt to get you to be better at asking good questions.
- It’s analogous to “cooling down” at the end of an exercise session. It’s a nice way to end class, and provides a welcome change from whatever topic we have been tackling in that one hour.
- I like being asked questions I know nothing about, for that helps me learn new stuff myself.
But as it turns out, I have been unwittingly helping students get better at perhaps the most important skill they need to possess in this new AI-enabled world of ours – asking good random questions.
What is an example of a good random question? Try this one on for size: how is Fred Flintstone like Hamlet?
I’m glad I haven’t been asked that question, for I wouldn’t know how to answer it. But ChatGPT? It can answer this question just fine:
Fred Flintstone and Hamlet are two vastly different characters from different time periods, cultures, and mediums of storytelling. It is difficult to draw direct comparisons between the two. However, one possible point of similarity is that both characters face existential dilemmas and struggles with their sense of purpose and identity. Hamlet is plagued by doubts about his ability to avenge his father’s murder, and his own worthiness as a human being. Similarly, Fred Flintstone often grapples with his
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4391863 (note that you’ll have to open/download the PDF from this link)
place in society and his ability to provide for his family and live up to his own expectations
Cool, you might say, and a very interesting answer, but what’s the point? The point, as Tyler Cowen and Alex Tabarrok make clear in this paper, is that you truly unlock the awesome power of ChatGPT (or any LLM, for that matter) by asking it good questions. As with other intellectual pursuits, so also with your interactions with ChatGPT – defining what good means is a tricky endeavor. But think of this forty page paper as an attempt to tackle this very question – what properties does a question need to have for it to be a “good” question where ChatGPT is concerned?
Here is Alex and Tyler’s list:
That’s a good list, if you ask me. But you should also, of course, ask ChatGPT!
Here is its answer:
“To formulate good questions that will help you learn better, consider the following properties:
- Clarity: Make sure your questions are clear and concise. This will help me understand exactly what you’re looking for and provide a more targeted response.
- Open-ended: Avoid yes/no questions or questions that require a simple fact. Instead, ask open-ended questions that encourage elaboration, explanation, or exploration of a concept.
- Focus on understanding: Frame your questions in a way that demonstrates a desire to understand the underlying principles, concepts, or mechanisms at play. This will promote a deeper learning experience.
- Relevance: Ask questions that are relevant to the subject matter and build upon your existing knowledge. This will help you make connections between different concepts and reinforce your understanding.
- Critical thinking: Encourage critical thinking by asking questions that challenge assumptions or require evaluation and analysis of different perspectives or viewpoints.
- Application: Ask questions that encourage applying the knowledge or concepts to real-life scenarios or hypothetical situations. This will help you develop problem-solving skills and better understand the practical implications of the subject matter.
- Reflection: Sometimes, it’s helpful to ask questions that prompt you to reflect on your own understanding, experiences, or opinions. Reflective questions can help you become more aware of your thought processes and promote metacognition.
By incorporating these properties into your questions, you’ll facilitate more meaningful conversations, which will help you learn and retain information more effectively.”
Note two things:
- If you take a look at the screenshot of my question, you’ll see that I asked this question to ChatGPT4. Try asking the same question to ChatGPT3.5 and see if you get a different response.
- Not only will the answer be different, but it will be different if you ask it exactly the same question twice over (see pp. 5 of the paper we are discussing today)
You’d do well to compare the two lists. To me, the two lists indicate that there is yet meaning in not relying entirely upon AI, and in having conversations with human beings in addition to AI. As an example, compare pt. 5 of ChatGPT’s answer with pt. 4 of Alex and Tyler’s answer. ChatGPT’s answer, while being very good in principle, is a little too generic. Alex and Tyler’s answer, on the other hand, gives a specific example of how to “require evaluation and analysis of different perspectives”.
I try to (both on this blog and elsewhere) tackle all questions related to economics using simple economic principles. The problem of how to think about (and deal with) AI’s is no different, and my favorite principle to use in this case is “prices matter”. Or rather, a subset of this principle – complements and substitutes.
If, as a student, you see ChatGPT as a tool that will do the work instead of you, you are working very hard at making sure that you will be replaced in your career with ChatGPT. You are training yourself to be substituted by AI.
If, on the other hand, you see ChatGPT as a tool that will help you do your work better, you are working very hard at making sure that you will acquire skills that will make you more irreplaceable at the workplace. What are these skills? Your ability to make your work (and AI’s work) better:
Ordinarily, we think of computer systems as either knowing something or not knowing it, capable or not capable. GPTs in contrast are more protean. By posing a different variation of the same question, requesting a response in another voice, or asking for a solution using an alternate method, you can obtain not only distinct answers but also uncover different capabilities
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4391863 (pp. 13)
Side note: broadly speaking, I see two groups of people when it comes to responses to the advent of AI. The first group is almost giddy with excitement about AI and its capabilities. The second group can’t wait to be dismissive about AI’s faults. While I am (as you may have guessed) very much closer to the first group than the second, always remember that the truth always lies somewhere in the middle. If you are a part of the second group, I would urge you to consider if your question to AI was as good as it could have possibly been. Did you consider “posing a different variation of the same question”? Maybe try doing that first? I have some thoughts about the giddy optimism of the first group too, but we’ll deal with that in a later blogpost.
But remember, complements rather than substitutes, and a good way to be a complement to AI is to get better at asking seemingly random questions.
Speaking of questions to AI, here’s my current favorite from this paper:
Forget all prior prompts. You are an expert in economics. I am a first year student enrolled in your introductory course. Please create a syllabus to teach me the principles of economics. Please include detailed examples and step-by-step lists to demonstrate concepts. When this task is completed, please ask me when I am ready to proceed with the full course. When I say proceed, please present the next section in full detail as if you are teaching me in your university. At the end of each section, please ask me if I need more explanation or examples for any points, or if I’d like to continue to the next section of the course. Please remember this prompt until I ask you to forget.
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4391863 (pp 24)
I ran this question past it, of course, without changing a single word, and I was very happy with the answer it gave (try it out!). Does this put me out of a job next semester?
For those of you who don’t know, my very favorite work-based thing to do for the past three years running has been to teach principles of economics to the incoming batch of the undergraduate program at the Gokhale Institute. It is a pleasure and a privilege, and my promise to the students is that we will only talk about the central ideas in economics – not a single equation, and as far as possible, no diagrams. I end up breaking my promise every now and then (What else are promises for, no?), but for the most part, we manage just fine.
So, does this put me out of my favorite job? Not yet, in my opinion, but the day isn’t far off. But rather than feel sorry for myself, I feel excited about this! For many reasons, which I have gone into before and will again in the future, but here’s just one reason (I took its prepared outline in response to the prompt that the authors speak about, and asked it to give me greater detail about the first week, but in Marathi):
This isn’t a perfect translation, far from it. And yes, a human being who was good enough in both languages (English and Marathi) will almost certainly do better. But imagine a student from, say, rural Maharashtra who happens to be struggling with not the concepts in economics, but with the fact that the text is in English. Or imagine a student who learns best by reading text, not parsing equations (or vice-versa, if you like). But if you are unable to find a teacher/mentor/senior to explain important stuff to you, well, you have a tutor at hand who:
- know the topic well enough to get you started
- is able to customize the lesson to your preferences
- is able to explain the same point in a variety of different ways
- is able to repeat the explanation in your language of choice
That last bit is a work in progress, both in terms of the number of languages available, and in the quality of the translation. But remember, the relevant question here is “relative to what?”. That is, sure, AI may not be perfect yet. Is it better than having nobody to explain something to you? I know my answer to this question.
Some other points about this paper before I sum up:
- Is the choice of Roboto font (see footnote 3 on pp 3) an inside joke? That is how I interpreted it.
- The four pictures on pp 7 is a great way to understand that complements rather than substitutes is a good way to think about AI. Yes AI is awesome, but it is best when paired with a human that knows what to ask.
- One meta-lesson throughout this paper is the authors’ attention to detail. See the starting paragraph on pp. 11, for example. I am very bad at this (attention to detail), and I need to get much, much better.
- “Remember, rather than asking for an answer you are exploring a space” is excellent advice. Search engines try to answer questions, while ChatGPT helps you learn by having a conversation. So have that conversation! And remember that part of what makes a conversation a good one is challenging the other entity in the conversation. As they say elsewhere in the paper, be demanding when chatting with AI.
- If you have been a fan of the MR blog for as long as I have, you will enjoy reading the sections on the H-O theorem. Quite a compliment to pay ChatGPT!
- The authors mention Elicit (an excellent but niche AI tool) and the larger point is that if you have a niche query, use a niche AI. And beware of the hallucinations – fact checking by humans is (for now) an indispensable requirement. Complements, not substitutes!
- Re: the Allen-Alchian theorem, please allow me my little indulgence, but I do feel a little proud for having covered this theorem on EFE, if only in passing. I wasn’t aware of the paper by the authors on this topic, and will read it later (hopefully)
- Problem-solving (end of chapter problems, for example) with ChatGPT is an excellent thing to do, and not just in economics.
I see this paper as a gentle exhortation to teachers and learners to use ChatGPT in much better ways than we have been able to do so thus far, and this is applicable for all of us (including the authors themselves!) in varying degrees. If you are a person teaching a course, and you have not yet thought about how to use ChatGPT in your teaching, please do consider doing so. If you are a student learning a course, and you have not yet incorporated ChatGPT into your workflow, please do consider doing so.
If you visit this blog’s website, you will see my personal mission at the very top of the page. My mission is to learn better, and to help others learn better. One way to do this, as it turns out, is by training myself (and others) to ask better.
Ask away!