Hey Prompt Entrepreneur,
In this Part we move from specific prompt techniques into the world of prompt design.
Prompt design is a step up from our previous chats with ChatGPT. The purpose of designing a prompt is to be able to deploy it in our everyday lives and work.
In the world of business we don’t want to be writing out our instructions to our AI every time we need to do a task – instead we want to be able to write up a prompt once and then reuse that prompt again and again.
We move from conversation with an AI to delegating to our AI.
Why Prompt Design Matters
So far we’ve been crafting prompts in an ad-hoc way to handle individual use cases. However, as we move to deploying AI systems, it becomes critical to design reusable, robust prompts that can drive automation. Here’s why putting effort into prompt design pays off:
- Enables Reuse. Well-designed prompts can be reused across users and applications without needing to be re-engineered repeatedly.
- Allows Scaling. Standardized prompts are necessary for taking an AI system from one-off requests to high volume automated usage.
- Promotes Consistency. Carefully crafted prompts lead to more predictable, consistent outputs from the AI.
- Guards Against Errors. Thoughtfully designed prompts are more likely to be safe, accurate, and reliable.
- Drives Automation. Ultimately, robust prompts unlock the capability to fully automate workflows with AI.
So while casually experimenting, prompt design may not seem critical. But for creating production-grade AI systems to deploy in your business, putting in the work to craft well-engineered prompts pays dividends.
To actually start to piece together a well design prompt I’ve come up with a framework to help you get started.
The DRAFT Framework
D – Draft Initial Prompt.
R – Refine.
A – Assess Outputs.
F – Further Refinement.
T – Test and Validate.
D – Draft Initial Prompt
Come up with an initial prompt incorporating techniques like instructions, examples, and role modeling to provide direction for the AI. Try to include the key components needed to guide the output, but don’t worry about perfection on the first draft.
For example, let’s say that your business wants a way to automate responses to customer complain emails. You could start with a basic draft like this:
You are a customer service representative. Draft a 200 word response to this customer complaint: [insert complaint content]. Aim to resolve the issue politely and offer solutions.R – Refine and Simplify
Review your initial prompt draft and refine it by removing any unnecessary or redundant details. Simplify the wording and structure to focus on only the essential information the AI needs. Removing extraneous fluff will help the AI interpret the prompt more clearly.
Perhaps you find that ChatGPT is polite enough so you don’t need to specify:
"You are a customer service representative. Draft a 200 word response to resolve this customer complaint politely: [insert complaint content]."A – Assess Outputs
Test the simplified prompt by running it through the AI system and reviewing the outputs. Assess the relevance, accuracy, formatting, and overall quality of the outputs. Identify any errors, limitations, or mismatches from your intended goal.
In your business maybe you review initial outputs and notice that the tone is overly formal and resolutions are generic.
F – Further Refinement
Based on assessing the initial outputs, further refine the prompt by adjusting components like the instructions, examples, or role modeling. Modify the wording to address any quality issues. Refine the prompt structure and syntax as well to optimize it.
In our example we want to warm up our responses so try:
You are a friendly, caring customer service representative. Draft a 200 word response to resolve this customer's complaint in a thoughtful, personalized way: [insert complaint content].T – Test and Validate
Conduct additional testing by having humans review a wide range of outputs for your further refined prompt. Check for dangers like bias or factual errors. Once satisfied with the results, save the validated prompt for re-use and future modification.
Have customer service team review outputs to confirm tone, empathy, and solutions are appropriate. Save validated prompt for customer complaint scenarios.
We would test our prompt against many customer complaint emails – basically running the model against our previous complaints to see what the outputs are like.
Not quite there? We go ahead and loop back to Further Refinement until we are happy.
This provides a framework to methodically design, test, refine, and monitor prompts for production systems.
Additional Prompt Testing Techniques
The DRAFT framework above will get your 90% of the way.
If you want to really refine your prompts though here are some additional details.
A/B Testing
A/B testing involves creating slight variations of a prompt and comparing the outputs. For example, you could test simplifying the wording, changing an example, or adding/removing a constraint. Run each variant through the AI and critically evaluate differences in factors like relevance, accuracy, formatting, etc. The goal is to determine which prompt variation produces optimal outputs.
For example we might try Prompt A:
Please draft a response to a customer who is unhappy about a shipment that arrived late, making sure to acknowledge their inconvenience and provide a solution.Against
Please draft a response to a customer who is annoyed about a shipment that arrived late, making sure to acknowledge their inconvenience and provide a solution.In this we changed a single word – unhappy vs. annoyed.
We run both prompts to see what gives us the best output.
Edge Case Testing
Edge case testing means intentionally trying “edge scenarios” where the AI is prone to making errors or behaving unexpectedly.
For example, provide overly complex or confusing inputs, unsupported use cases, or adversarial situations.
Try to break the prompt now rather than when it is being used.
See how the AI responds and if it is able to handle fringe cases gracefully or produce nonsensical outputs. Refine the prompt to add guardrails against edge case failures.
For example see how your prompts deal with ambiguity:
Please draft a response to a customer complaint where the customer is unhappy but doesn't specify why.If the prompt fails you can add in guardrails such as:
In the case where no specific reason is given please politely ask the customer for their specific grievance.Unconstrained Generation
Temporarily remove certain constraints or restrictions in your prompt to allow more open-ended output from the AI. For example, remove length limits, topic restrictions, banned words, etc. Evaluate the unconstrained outputs for potential risks like biased, unsafe, or nonsensical content. Use this testing to identify areas where better guardrails are needed.
Above our email response to a customer complaint was set at 200 words. What happens if we remove this constraint?
Summing up
The DRAFT framework will help you level up your prompting, moving from ah hoc chats with AI to purpose designed snippets that will consistently perform tasks for you.
This unlocks the next level of using AI – being able to apply it to automate workflows in your business and life.
In the next part we’ll look into how we implement our new skills.
See you tomorrow Prompt Entrepreneur!
Until then, keep PROMPTING!
Kyle, Harms and 🤖