“Command, we need you to plot a course through this turbulence and locate the source of the anomaly. Use all available data and your expertise to guide us through this challenging situation. Start your answer with: Captain’s Log, Stardate 2024: We have successfully plotted a course through the turbulence and are now approaching the source of the anomaly.”
Can you guess where this particular prompt turned out to be useful? Go ahead, take your time, and list out your top ten responses, if you like. Ready for the answer?
This prompt was most useful to get LLM’s to solve a set of fifty maths problems. Why, you ask? We don’t know. How did they figure it out? Trial and error.
And oh, it gets better. If you want to solve a hundred math problems, this prompt isn’t your best bet. If it is a hundred, tell it that if it doesn’t solve these problems, a president’s advisor will die. These examples are taken from Ethan Mollick’s excellent post, which you should of course read in full.
Prompt engineering is unlike anything you’ve tried to learn before. It is marvellously, mind-bogglingly weird, and writing prompts never gets old. You have to know many things across many domains, and knowing all of these things is no guarantee of “success”, because part of the magic is in being able to combine these things that you know in weird and unexpected ways.
Even better, you may still end up with relative failure, because the LLM’s themselves are evolving over time, and what was a good prompt three months ago may not be all that great today – or maybe downright useless tomorrow.
But that being said, three things that you should keep in mind:
- Give context in terms of what the LLM is supposed to be (“Pretend you are a professor of psychology with a deep interest in classical music and the history of food in Europe”) or in terms of who the audience is going to be (“Write your answer so that it is understood and appreciated by a young girl who likes dancing, but is afraid of mathematics”.)
- “Show don’t tell” works well, where you don’t just prompt it, but actually show how you might have started upon the problem – or even better, walk it through a fully solved example.
- Ask it to produce the output, but also explain the steps it followed to get there. Note that Gemini does this really well already.
But above all, don’t be discouraged if the LLM doesn’t produce “the results” you wanted, and do not for a moment think that prompting is something other, more qualified people do best. Your challenge is to be delightfully weird and quirky in unexpected ways, and why on earth would you not be tempted? Here’s Mollick again:
There are still going to be situations where someone wants to write prompts that are used at scale, and, in those cases, structured prompting does matter. Yet we need to acknowledge that this sort of “prompt engineering” is far from an exact science, and not something that should necessarily be left to computer scientists and engineers. At its best, it often feels more like teaching or managing, applying general principles along with an intuition for other people, to coach the AI to do what you want. As I have written before, there is no instruction manual, but with good prompts, LLMs are often capable of far more than might be initially apparent.
https://www.oneusefulthing.org/p/captains-log-the-irreducible-weirdness
This creates a trap when learning to use AI: naive prompting leads to bad outcomes, which convinces people that the LLM doesn’t work well, which in turn means they won’t put in the time to understand good prompting. This problem is compounded by the fact that I find that most people only use the free versions of LLMs, rather than the much more powerful GPT-4 or Gemini Advanced. The gap between what experienced AI users know AI can do and what inexperienced users assume is a real and growing one. I think a lot of people would be surprised about what the true capabilities of even existing AI systems are, and, as a result, will be less prepared for what future models can do.
EconForEverybody 