“Command, we need you to plot a course through this turbulence and locate the source of the anomaly. Use all available data and your expertise to guide us through this challenging situation. Start your answer with: Captain’s Log, Stardate 2024: We have successfully plotted a course through the turbulence and are now approaching the source of the anomaly.”
Can you guess where this particular prompt turned out to be useful? Go ahead, take your time, and list out your top ten responses, if you like. Ready for the answer?
This prompt was most useful to get LLM’s to solve a set of fifty maths problems. Why, you ask? We don’t know. How did they figure it out? Trial and error.
And oh, it gets better. If you want to solve a hundred math problems, this prompt isn’t your best bet. If it is a hundred, tell it that if it doesn’t solve these problems, a president’s advisor will die. These examples are taken from Ethan Mollick’s excellent post, which you should of course read in full.
Prompt engineering is unlike anything you’ve tried to learn before. It is marvellously, mind-bogglingly weird, and writing prompts never gets old. You have to know many things across many domains, and knowing all of these things is no guarantee of “success”, because part of the magic is in being able to combine these things that you know in weird and unexpected ways.
Even better, you may still end up with relative failure, because the LLM’s themselves are evolving over time, and what was a good prompt three months ago may not be all that great today – or maybe downright useless tomorrow.
But that being said, three things that you should keep in mind:
Give context in terms of what the LLM is supposed to be (“Pretend you are a professor of psychology with a deep interest in classical music and the history of food in Europe”) or in terms of who the audience is going to be (“Write your answer so that it is understood and appreciated by a young girl who likes dancing, but is afraid of mathematics”.)
“Show don’t tell” works well, where you don’t just prompt it, but actually show how you might have started upon the problem – or even better, walk it through a fully solved example.
Ask it to produce the output, but also explain the steps it followed to get there. Note that Gemini does this really well already.
But above all, don’t be discouraged if the LLM doesn’t produce “the results” you wanted, and do not for a moment think that prompting is something other, more qualified people do best. Your challenge is to be delightfully weird and quirky in unexpected ways, and why on earth would you not be tempted? Here’s Mollick again:
There are still going to be situations where someone wants to write prompts that are used at scale, and, in those cases, structured prompting does matter. Yet we need to acknowledge that this sort of “prompt engineering” is far from an exact science, and not something that should necessarily be left to computer scientists and engineers. At its best, it often feels more like teaching or managing, applying general principles along with an intuition for other people, to coach the AI to do what you want. As I have written before, there is no instruction manual, but with good prompts, LLMs are often capable of far more than might be initially apparent. This creates a trap when learning to use AI: naive prompting leads to bad outcomes, which convinces people that the LLM doesn’t work well, which in turn means they won’t put in the time to understand good prompting. This problem is compounded by the fact that I find that most people only use the free versions of LLMs, rather than the much more powerful GPT-4 or Gemini Advanced. The gap between what experienced AI users know AI can do and what inexperienced users assume is a real and growing one. I think a lot of people would be surprised about what the true capabilities of even existing AI systems are, and, as a result, will be less prepared for what future models can do.
You must have heard of the drunk who was searching for his keys under a streetlight. When asked why he is searching here, rather than the place where he had dropped the keys, the drunk replies that he can see well over here.
Haha and all that.
Here’s Kenneth Arrow writing about a person you likely have not heard of, GLS Shackle. Arrow is writing about why people aren’t interested in reading Shackle anymore:
The reason for the current lack of interest is probably not any denial that Shackle’s position is fundamentally correct; it is the absence of the analytic tools needed to make the exceptional approach capable of generating operationally meaningful conclusions.
What was GLS Shackle’s position, and who was GLS Shackle?
GLS Shackle was an economist. Wikipedia tells us that he started work on a PhD under the supervision of Hayek, but that he later switched to “an interpretation of Keynes’s General Theory of Employment, Interest and Money”. Because we know that the truth lies somewhere in the middle, this is a most wonderful thing, of course. Note too that there is a correct way to handle a situation such as this, and Hayek was more than up to the task:
A student of Hayek’s at the London School of Economics in the 1930s, Shackle renounced his early Hayekian views and the doctoral dissertation on capital theory that he had already started writing under Hayek’s supervision, after hearing a lecture by Joan Robinson in 1935 about the new theory of income and employment that Keynes was then in the final stages of writing up to be published the following year as The General Theory of Employment, Interest and Money. When Shackle, with considerable embarrassment, had to face Hayek to inform him that he could not finish the dissertation that he had started, no longer believing in what he had written, and having been converted to Keynes’s new theory. After hearing that Shackle was planning to find a new advisor under whom to write a new dissertation on another topic, Hayek, in a gesture of extraordinary magnanimity, responded that of course Shackle was free to write on whatever topic he desired, and that he would be happy to continue to serve as Shackle’s advisor regardless of the topic Shackle chose.
And what was his position? This is not an easy question to answer, and I will admit that I am not entirely sure of the correct answer. But given that disclaimer, here is the best I can do:
You can either reject rationality or time
What does this mean, and why does it matter?
A rational-expectations model allows for stochastic variables (e.g., will it be rainy or sunny two weeks from tomorrow), but those variables are assumed to be drawn from distributions known by the agents, who can also correctly anticipate the future prices conditional on any realization (at a precisely known future moment in time) of a random variable. Thus, all outcomes correspond to expectations conditional on all future realizations of random variables; there are no surprises and no regrets. For a model to be correct and determinate in this sense, it must have accounted fully for all the non-random factors that could affect outcomes. If any important variable(s) were left out, the predictions of the model could not be correct. In other words, unless the model is properly specified, all causal factors having been identified and accounted for, the model will not generate correct predictions for all future states and all possible realizations of random variables. And unless the agents in the model can predict prices as accurately as the fully determined model can predict them, the model will not unfold through time on an equilibrium time path. This capability of forecasting future prices contingent on the realization of all random variables affecting the actual course of the model through time, is called rational expectations, which differs from perfect foresight only in being unable to predict in advance the realizations of the random variables. But all prices conditional on those realizations are correctly expected
I wouldn’t blame you if your eyes glazed over while reading this. But bear with me, and let’s go over this slowly:
Imagine that you are a farmer, and you are going to harvest strawberries two weeks from now.
If it rains at or around the time of the harvest, your harvest is going to be destroyed. If it doesn’t rain, you will have strawberries to sell. Should you assume that you will have a harvest ready to sell, or not?
Imagine that your best friend has a shop that sells smartphones, and you’ve told him that you will buy a smartphone from his shop, once you’ve sold your strawberries. Should he assume the sale of a smartphone to you, or not?
The strawberries from your farm are usually purchased by a jam manufacturing company. This company has, for the first time ever, won a contract to sell its famous strawberry jam to Walmart. They got this contract because of the quality of your strawberries. No harvest on your farm, no sale of their jam to Walmart.
This firm has promised temporary employment to locals to make this jam, who in turn have promised their families a trip to their hometown once the jam manufacturing company pays them.
Toy manufacturers in that hometown are anticipating a rise in sales when these families come visiting.
We don’t know if it will rain in two weeks or not. Nobody does. But should we assume that we can forecast rain with x% probability?
And should we therefore assume that we know today the prices of smartphones, the wage-rate for hiring workers to make jam, the price of a ticket to a village nearby, and the prices of toys in that village – all of these two weeks from now?
Now read the last two sentences in that excerpt above: “This capability of forecasting future prices contingent on the realization of all random variables affecting the actual course of the model through time, is called rational expectations, which differs from perfect foresight only in being unable to predict in advance the realizations of the random variables. But all prices conditional on those realizations are correctly expected”
GLS Shackle was saying that there is no such thing as rational expectations, because, well, time. That is, if you accept rational expectations as a feature of your model, you must reject time. And if you accept time, you must reject rational expectations.
So, dear reader, do you accept the existence of time, or not?
But what does the “existence of time” mean, in the context of economics?
Note that I am going to keep things as simple as possible, both for your sake and mine. And the simplest possible answer to this question is this:
Acknowledging the existence of time necessarily implies that the future is always, and by definition, unknowable
You have two choices: to continue reading from here on in, or to stop reading at this point. You have created your future by choosing one of these two courses of action. And until you make that choice, your future is unknown.
If, after you stopped reading, you walked into a store and purchased a jar of strawberry jam, you changed the future of the store owner, the strawberry jam manufacturer, the part time employee of the strawberry jam manufacturer, the kid of the part time employee and the toy manufacturer. If you didn’t stop reading, and therefore ran out of time to buy that jar, you changed their future too! And of course, if it rains (rained?) in the next two weeks, there might not be any strawberry jam to buy in the first place! Oh, and of course you could sprain your ankle while stepping into that store. Or <insert random event of your choice here>.
We just don’t know what our future holds, and our future is being shaped and fashioned by the choices made by million of other people – in much the same way that their futures are being shaped and fashioned by the choices made by you.
Bottomline?
Acknowledging the existence of time necessarily implies that the future is always, and by definition, unknowable
Again, GLS Shackle was saying that there is no such thing as rational expectations, because, well, time. That is, because the future is inherently and by definition unknowable – not predictable, not modelable – there can be no such thing as rational expectations.
Is there any point, then, to building a rational expectations model?
If you say no, you are a newly minted member of Team Shackle. If, on the other hand, you say yes – well, I hope you find those keys of yours under this streetlight.
“Munch would have probably seen any marks from this period of the painting’s life as part of its artistic development. He wanted people to see how his works evolved and changed over their lifetime, and saw any damage they incurred along the way as a natural process, even leaving artworks unprotected outdoors and in his studio, stating ‘it does them good to fend for themselves’.”
..
..
I cannot for the life of me remember how I chanced upon this link – all that I remember is that it came out of an interesting Twitter thread. 10 factoids about The Scream.
..
..
“It’s called the “dinner party problem”: A table of four or fewer people may happily converse as one, but a party of five or more will splinter fairly quickly into separate conversations of two or three four people each. What is it about the number four?”
..
..
It really should be called the panel discussion problem. The conclusion to the short article deserves to be highlighted!
..
..
“It’s possible our brains evolved to manage only the conversations in which we have a chance of swaying the group to our side. Otherwise, what’s the point of talking?”
..
..
I’ll happily admit to the fact that the math is way beyond my capabilities – but it made for enjoyable viewing, if nothing else. The Mandelbulb, or the 3D version of the Mandelbrot set. This is via Navin Kabra, who should immediately be followed on Twitter.
..
..
“Are Indigenous and Western systems of knowledge categorically antithetical? Or do they offer multiple points of entry into knowledge of the world, past and present?”
..
.. A very interesting article in the Smithsonian on what is knowledge, and how is to be gleaned, understood and used.
..
..
EconForEverybody 